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PREFACE 


The  Integrated  Technical  Information  for  the  Air  Logistics  Centers  (ITI-ALC)  Final 
Documentation  presented  in  this  report  was  prepared  under  Government  contract  F41624-94-C- 
5021  in  accordance  with  Contract  Data  Requirements  List  (CDRL)  sequence  number  A009.  This 
report  reflects  the  work  accomplished  by  Systems  Research  and  Applications  (SRA)  Corporation 
and  ARINC  Research  Corporation,  and  sponsored  by  Armstrong  Laboratory/Logistics  Research 
Division,  Operational  Logistics  Branch  (AL/HRGO),  Wright-Patterson  Air  Force  Base  (AFB), 
OH.  This  report  was  developed  under  the  leadership  of  Ms.  Barbara  Masqueilier,  the  AL/HRGO 
Program  Manager,  and  Mr.  Ron  Kelly,  the  SRA  Corporation  Principal  Investigator  for  the  ITI- 
ALC  program. 

During  this  first  phase  of  the  ITI-ALC  program,  the  ITI-ALC  team  visited  the  Air  Logistics 
Centers  (ALCs)  to  collect  data  required  to  document  the  current  Programmed  Depot  Maintenance 
(PDM)  process,  to  validate  our  understanding  of  the  process,  and  to  review  and  enhance  the 
improvement  concepts  for  the  PDM  process  and  ITI-ALC  system.  This  user  involvement  played 
a  significant  role  to  ensure  that  the  improved  PDM  capability  will  result  in  sigmficant  benefits  to 
depot  personnel.  The  ITI-ALC  team  expresses  its  appreciation  for  the  valuable  cooperation, 
contributions,  and  support  received  firom  the  personnel  within  the  following  depots: 

•  Ogden  Air  Logistic  Center  (00-ALC) 

•  Oklahoma  Air  Logistic  Center  (OC-ALC) 

•  Sacramento  Air  Logistic  Center  (SM-ALC) 

•  San  Antonio  Air  Logistic  Center  (SA-ALC) 

•  Wamer-Robins  Air  Logistic  Center  (WR-ALC) 
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1.  SUMMARY 


The  eve  of  the  21st  century  marks  more  than  a  chronological  milestone.  Converging  changes  in 
technology  and  economics,  and  fundamental  restructing  and  downsizing  of  the  military  call  for 
fresh  thinking  about  how  to  harness  information  technology  to  provide  tangible  value  to 
organizations  and  users.  The  challenge  is  to  adapt  organizations  and  processes  to  rapidly 
changing  technologies  and  methodologies  to  achieve  greater  effectiveness  and  quality  at  reduced 
cost.  Organizations  that  master  change  will  realize  their  goals,  while  those  who  fail  to  reengineer 
their  policies  and  practices  will  diminish  in  stature  and  gradually  fade  away . 

The  ALCs  are  poised  on  the  forward  edge  of  military  readiness.  Budget  realities  demand  that 
older  and  often  heavily  modified  aircraft  remain  in  the  inventory  longer;  thus  increasing  the 
importance  of  cost-effective,  improved  depot  maintenance.  These  improvements  require  that 
better,  more  timely,  and  seamlessly  integrated  information  -  information  currently  resident  in 
numerous  systems  --  be  made  available  to  the  depot-level  mechamcs,  managers,  and  planners. 

The  budget  austerity  that  spawned  the  current  emphasis  on  functional  process  improvements  and 
reengineered  business  practices  is  not  likely  to  abate.  Managers  in  every  orgamzation  must 
objectively  rethink  their  current  processes  and  challenge  the  status  quo.  Merely  injecting 
technology  without  improving  the  underlying  processes  yields  marginal,  short-term 
improvements  -  not  the  type  of  fundamental  breakthrou^  that  are  imperative  if  the  ALCs  are 
to  do  more  for  less.  Only  by  reengineering  its  operations  can  the  Air  Force  realize  the  hoped-for 
productivity  and  quality  improvements  that  are  needed  to  meet  its  mission. 

The  in-ALC  program  was  established  to  address  these  objectives  of  integrating  and  delivering 
the  information  required  in  the  depot  maintenance  process.  Specifically,  the  fTI-ALC  initiative 
focused  on  developing  an  improved  process  definition  for  the  PDM  aspect  of  depot  maintenance 
with  a  limited  look  at  component  repair,  and  the  requirements  and  specification  for  the  ITI-ALC 
system  to  automate  and  support  the  unplementation  of  the  improved  process.  This  improved 
process  and  the  ITI-ALC  system  will  help  to  standardize  and  integrate  maintenance  processes 
and  information  not  only  within  a  depot  but  also  across  the  depots. 

The  improved  process  and  the  applications  of  technologies  for  the  depot  maintenance  process 
was  developed  using  a  structured,  user-focused  methodology  that  included  active  participation  by 
maintenance  persoimel  from  all  five  ALCs.  Through  this  active  participation,  mfiirmation  about 
the  current  process  was  collected,  along  with  user-identified  problems  and  improvement  ideas. 
This  user-specified  information  was  analyz;^  and  aggregated  into  a  set  of  "AS-IS"  models  which 
represented  a  unified  view  of  the  current  depot  maintenance  processes.  This  "AS-IS"  view  was 
validated  by  the  users  to  ensure  a  solid  foundation  on  which  to  build  the  improved  "TO-BE" 
concept  for  the  depot  maintenance  process. 

Analysis  of  the  "AS-IS"  depot  maintenance  process  identified  problem  areas  and  potential 
improvement  opportunities.  Combining  these  identified  problem  areas  and  potential 
improvement  opportunities  with  ftiose  suggested  by  the  users,  a  set  of  15  Business  Process 
Improvements  (BPIs)  were  identified  that  impacted  a  range  of  depot  maintenance  characteristics. 
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These  characteristics  listed  in  Figure  1-1,  include  role  changes  for  personnel,  data  changes  based 
on  the  commonaKly  of  data  within  as  well  as  across  the  ALCs,  process  changes  to  streamline 
depot  maintenance,  organuational  changes  to  focus  attention  on  the  ultimate  goal  of  getting 
aircraft  repaired,  and  policy  changes  that  support  rather  than  hinder  die  improved  depot 
maintenance  process  per  the  performance  targets. 
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Figure  1-1.  The  BPIs  Translate  into  Operational  Savings 


With  the  requirements  and  specifications  for  the  improved  process  established,  Ae  requirements 
for  the  supporting  ITI-ALC  system  were  developed,  along  with  its  high-level  design. 

A  business  case  formed  the  basis  to  quantify  the  improvement  benefits.  The  values  of  the 
benefits  were  based  on  various  levels  of  the  fTI-ALC  system  implementation.  These  levels  of 
implementation  extended  from  implementing  the  improved  process  without  the  m-ALC  system 
to  full  implementation  of  the  ITI-ALC  system  to  support  the  improved  process.  As  represented 
in  Figure  1-1,  full  implementation  of  the  improved  PDM  process  and  the  supporting  ITI-ALC 
system  providing  realistic  savings  of  40%  in  operational  expenses  and  30 /o  in  aircraft 
maintenance  flow  days. 


The  stmetured,  user-focused  methodology  appUed  to  the  IH-ALC  program  has  been  developed 
and  improved  over  a  number  of  years  and  applications.  During  its  application  on  the  ITI-ALC 
program,  additional  lessons  were  learned  with  respect  to  its  strengths,  weaknesses,  and 
improvement  ideas  for  its  next  applications.  The  following  list  summarizes  possible 
recommendations  and/or  lessons  learned  that  were  identified  durii^  Phase  I  of  the  IH-ALC 
program.  Section  3  provides  more  detail  on  possible  recommendations  and/or  lessons  learned 
that  were  identified  as  part  of  each  project  entity;  such  as,  the  various  models  and  formal 
deliverables.  This  list  of  possible  recommendations  and/or  lessons  learned  is  repeated  in 
Appendix  C. 


•  Significant  preparation  for  the  data  collection  trips  is  critical. 

User  acceptance  is  raitical  to  any  system  development  effort.  Because  the  data  collection 
trips  were  the  first  technical  interface  with  the  users  and  because  humans  have  a  tendency  to 
let  "first  impressions  be  lasting  impressions,"  the  data  collection  trips  provided  the  first  and 
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best  opportunity  to  gain  the  users'  acceptance  of  and  involvement  in  the  ip-ALC  program. 
The  data  collection  trips  were  well  planned  as  determined  by  the  positive  feedback  an 
involvement  received  from  the  ALC  personnel. 

•  The  model  development  order  must  be  such  that  the  benefits  of  their  integration  is  fully 
utilized. 

The  methodology  applied  to  the  ITI-ALC  program  was  based  on  the  development  and 
analysis  of  an  integrated  set  of  models  and  reports  documenting  the  requirements 
specifications,  design,  and  benefits  predicted  from  implementing  the  improved  process  and 
the  ITI-ALC  system.  The  models  developed  for  this  methodology  were  the  following: 

-  “AS-IS”  and  “TO-BE”  Functional  Models, 

-  “AS-IS”  and  “TO-BE”  Data  Models, 

-  “AS-IS”  and  “TO-BE”  Process  Models, 

-  “AS-IS”  and  “TO-BE”  Simulation  Models,  and 

-  System  Model. 

The  reports  developed  were  the  ITI-ALC  Business  Case,  System/Segment  Specifi^tion 
(SSS),  and  the  System/Segment  Design  Document  (SSDD).  The  development  order  of  these 
items  was  good,  but  minor  adjustments  would  have  improved  the  effective  performance  of 
the  ITI-ALC  program.  Of  specific  importance  was  the  development  and  analysis  of  the 
process  flow  and  simulation  models.  Having  established  the  requirements  for  Aese  models 
and  analyses  earlier  in  the  program  would  have  reduced  some  of  the  duplicative  data 

collection  efforts. 


•  Validation  difficulty  varied  among  the  various  artifacts  developed. 

The  various  "AS-IS"  and  "TO-BE"  models,  which  formed  the  foundation  for  the  program, 
required  significant  time  and  effort  for  validation  by  the  functional  experts  and  users.  These 
validation  efforts  were  implemented  using  the  readership  cycle  and  on-site  walktooughs. 
While  the  review  results  were  sufficient  for  validation,  the  validation  of  the  SSS  handled  as  a 
workshop  away  from  the  ALC  worksite  proved  to  be  more  productive.  Therefore,  the  use  of 
similar  workshops  for  the  "AS-IS"  and  “TO-BE”  models  would  be  recommended. 

•  Data  flow  diagrams  are  an  excellent  way  to  represent  the  system  model. 

The  top-down  approach  supported  by  data  flow  diagrams  provided  gradud,  confrolled 
concept  of  operation  for  the  ITI-ALC  system.  This  approach  also  provided  a  direct  link  of 
information  from  the  functional  and  data  models,  diereby,  supporting  the  requirement  tor  the 
traceability  of  information  through  the  models.  While  the  concept  of  usmg  data  flow 
diagrams  was  effective,  the  System  Architecture  tool  used  to  developed  these  diagrams  had 
some  short  coinings  that  need  to  be  corrected  to  better  support  data  flow  diagram 
development. 
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The  concept  of  developing  and  using  process  flow  diagrams  via  the  Integrated 
DEFinition  (IDEF3)  notation  and  simulation  via  WITNESS  was  value  added  to  the 
overall  ITI-ALC  program. 

The  development  of  the  process  flow  diagrams  was  facilitated  by  the  existence  of  the 
functional  models  while  the  development  of  the  process  flow  models  helped  to  increase  the 
accuracy  and  completeness  of  the  ftmctional  models.  The  process  model  provided  an 
effective  structure  to  identify  and  collect  die  performance  information  required  from 
simulation  and  provided  an  effective  structure  for  developing  and  depicting  the  simulations. 

The  tools  used  to  implement  the  process  flow  model  and  the  translation  to  simulation 
are  not  yet  mature. 

The  translation  capability  from  ProSim™  to  WITNESS®  was  not  complete.  The  transfer 
from  ProSim  to  WITNESS  required  a  significant  amount  of  additional  data  and  data 
manipulation  in  WITNESS  before  the  process  flow  could  be  exercised  using  WITNESS. 
During  the  TTI-ALC  program,  this  ineffective  transfer  of  information  was  overcome  by 
developing  an  in-depth  understanding  of  the  transfer  requirements  and  accomplishing  much 
of  it  manually. 

The  translation  from  the  process  description  to  the  simulation  was  one-directional.  The 
intent  of  IDEF3,  as  implemented  via  ProSim,  is  to  collect  process  flow  and  performmce 
information  so  as  to  facilitate  the  development  and  execution  of  a  simulation  model  within 
WITNESS.  However,  because  significant  effort  is  required  to  complete  the  simulation  model 
in  WITNESS,  the  value  of  the  IDEF3  model  via  ProSim  is  significantly  reduced  once  the  first 
translation  is  accomplished.  During  the  TTI-ALC  program,  this  situation  was  addressed  by 
maintaming  the  IDEF3/ProSim  representation  for  display  of  the  network  while  adjusting  the 
model  within  WITNESS  as  needed  for  the  simulation. 

The  process  model  notation,  as  represented  by  ProSim,  was  not  an  effective  presentation 
vehicle  for  two  reasons.  First,  the  information  presented  on  a  process  flow  network  was  very 
limited,  making  it  a  labor  intensive  effort  to  read  and  analyze  a  process  flow.  Second,  the 
amount  of  readable  information  printed  on  one  page  was  limited,  forcing  the  model  to  be 
developed  in  short  segments.  Connecting  the  short  segments  to  form  and  effectively  depict  a 
larger  process  was  difficult.  This  difficulty  was  reduced  by  using  the  fonctiorial  model  and 
performance  data  sheets  to  supplement  the  use  of  the  process  flow  represented  via  ProSim. 

An  integrated  team  effort  is  required  to  maximize  the  quality  of  the  BPIs  and  the 
products  developed  throughout  the  program. 

Throughout  the  performance  of  the  ITI-ALC  program  communication  among  the  team 
members  was  important  to  ensure  a  unified  understanding  of  the  "AS-IS"  and  "TO-BE  depot 
maintenance  processes,  and  was  critical  during  the  development  of  the  System  Model  (SM), 
SSS,  and  SSDD.  Even  though  the  SM,  SSS,  and  SSDD  developments  are  based  on  the  "TO- 
BE"  models,  the  interpretation  of  the  models  can  vary  slightly,  causing  potential 
inconsistencies  among  these  artifacts.  Close  interactions  among  the  developers  of  all  the 
program  artifacts,  and  especially  the  SM,  SSS,  and  SSDD  maximized  the  effectiveness  of  the 
ITI-ALC  system  requirements,  and  high-level  design. 
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1.1  INTRODUCTION 

The  objective  of  the  ITI-ALC  program  was  to  improve  the  efficiency  and  effectiveness  of  the  Air 
Force's  depot  maintenance  process,  with  specific  emphasis  on  PDM  operations  and  with  a  l^ted 
look  at  component  repair.  This  was  accomplished  by  streamlining  the  process  and  developing  an 
ITI-ALC  integrated  system  concept  implementable  with  technologies  that  will  automate  the 
streamlined  process  to  standardize,  integrate,  and  make  information  easily  accessible  to  the  ALC 
personnel  through  a  single  entry  source.  This  integration  of  information  included  interfacing 
with  many  independent  sources  of  information  such  as  engineering  drawings,  maintenance  plans, 
manufacturing  specifications,  technical  orders,  and  dynamic  diagnostics.  When  implemented, 
the  results  of  the  ITI-ALC  program  will  reduce  flow  days,  improve  quality,  reduce  operating 
costs,  and  improve  mechanic  performance. 

The  ITI-ALC  program  was  accomplished  through  the  application  of  a  structured,  user-focused 
methodology  that  supported  the  effective  collection,  integration,  and  analysis  of  the  user 
information.  The  approach  produced  effective  and  practical  BPI  concepts  for  the  streamlined 
PDM  process  and  the  requirements  for  the  supporting  ITI-ALC  system. 

During  the  ITI-ALC  program,  the  team  visited  all  five  of  the  Air  Force's  ALCs,  as  well  as  some 
commercial  depot  centers,  to  support  the  data  collection,  model  development,  process  and  system 
specification,  and  validation  process.  These  direct  interactions  with  the  users  mcluded  mitial 
interviews  with  139  users  and  follow-up  interviews  with  38  users  for  a  total  of  111  data 
collection  interviews.  Furthermore,  the  interaction  began  in  the  interviews  was  continued  by 
model  readership  (103  total  reviews),  validation  meetings  (29  user  representatives),  and  a  user 
workshop  (10  user  representatives). 

This  report  describes  the  methodology  that  was  applied  to  successfully  satisfy  the  requirements 
of  the  ITI-ALC  program  along  with  possible  recommendations  and/or  lessons  learned  during  the 
performance  of  the  ITI-ALC  program.  To  facilitate  the  reading  of  this  report,  it  is  divided  into 
sections  containing  two  levels  of  detail.  Section  2  presents  an  overview  of  the  methodology. 
Section  3  presents  more  details  about  each  aspect  of  the  me&odology  along  with  possiWe 
recommendations  and/or  lessons  learned  related  to  the  major  stages  that  make  up  the 
methodology,  and  Section  4  presents  conclusions  and  recommendations. 

1.2  BACKGROUND 

Depot  maintenance  is  responsible  for  all  the  scheduled  and  unscheduled  maintenance  of  airc^ 
other  aerospace  vehicles,  and  associated  systems  and  components,  such  as  engines  and  landmg 
gears.  An  effective  depot  maintenance  process  provides  the  operating  organizations  mth  sufficient 
quantities  of  aircraft  and  serviceable  items  to  train  aircrews  in  peacetime  ^d  to  fly  missions  m  toe 
event  of  war.  While  recent  improvements  in  system  reliability  are  reducing  toe  time  required  for 
an  item  to  come  into  toe  depot  process,  budgets  for  new  systems,  sp^es,  and  mechanic  traimng 
are  decreasing.  Therefore,  toe  requirement  to  use  more  effective  and  efficient  ways  to 
accomplish  toe  depot  maintenance  process  is  more  critical  today  than  ever  before. 
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Computer  and  information  technology  advances  have  driven  many  projects  to  improve 
information  access  within  and  among  maintenance  organizations.  Other  projects  have  been 
directed  to  improve  tools  and  maintenance  aids  for  mechamcs.  Until  now,  however,  no  attempt 
has  been  maHp.  to  integrate  the  available  information,  tools,  and  aids  for  the  mechamc.  The  ITI- 
ALC  program  goal  was  to  integrate  tiiese  elements  by  focusing  on  the  maintenmce  mechanic’s 
needs  as  the  most  important  aspect  of  this  integration  process.  This  integration 
accomplished  through  the  design  of  a  streamlined  depot  maintenance  process  along  with  th^TI- 
ALC  ^stem  to  facilitate  the  performance  of  the  streamlined  depot  maintenance  process.  Thus, 
the  ultimate  value  of  the  ITI-ALC  system  and  its  acceptance  by  the  end-user  is  directly  linked  to 
the  m-ALC  program’s  ability  to  achieve  measurable  performance  improvements  at  the  mechamc 
level.  This  user  focus  is  the  foundation  for  the  systematic  approach  used  by  the  m-ALC  team  to 
develop  an  ITI-ALC  architecture. 

The  overall  ITI-ALC  program  is  represented  in  Figure  1-2,  with  the  goal  of  the  first  phase  being 
to  develop  the  specifications  and  design  for  the  m-ALC  system  and  the  second  phase  to  develop 
a  demonstration  or  "proof  of  concept"  system.  The  demonstration  system  will  be  evaluated  with 
respect  to  the  improvement  predictions  developed  for  the  design.  Any  ^rto^ce 
descrepancies  are  used  to  refine  the  design  and  the  implementation  until  the  acceptable  ITI-ALC 

system  is  ready. 

NOTE:  This  final  report  only  pertains  to  Phase  I  of  the  overall  IH-ALC  program. 


Figure  1-2.  An  Overview  of  the  Total  ITI-ALC  Program  Concept 


2.  METHODS  AND  ASSUMPTIONS 


2.1  METHODS 

The  ITI-ALC  program  was  performed  successfully  through  the  application  of  a  user-focused 
methodology,  as  represented  in  Figure  2-1.  Working  closely  with  the  users  helped  to  understand 
the  work  performed  within  the  five  ALCs,  provided  a  solid  foundation  for  the  BPI  analysis  effort, 
and  provided  the  basis  to  present  and  gain  acceptance  of  the  improvement  concepts  as  they  were 
developed. 


Figure  2-1.  The  ITI-ALC  Program  Methodology  is  User  Focused 


To  obtain  data  needed  to  describe  the  current  depot  maintenance  process,  a  data  collection 
process  consisting  primarily  of  on-site  interviews  was  implemented  at  each  of  the  five  ALCs.  to 
addition,  user  ideas  were  also  collected  that  identified  what  they  believed  was  wrong  with  the 
process  and  how  the  process  could  be  improved. 

To  effectively  minimize  the  data  collection  effort  required  that  a  well  defined  set  of  data 
requirements  be  established.  These  data  requirements  were  identified  included  activity-onented 
interface-oriented  data,  and  operations-oriented  data.  The  activity-oriented  data  mcluded 
activity  definition  information,  inputs  and  outputs  for  the  activities,  and  regulations  and 
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guidelines  that  defined  the  process  of  implementing  the  activities.  Interface-oriented  information 
included  manual  and  automated  interfaces  among  the  personnel  and  organi^tions  within  the 
depot,  and  between  the  depot  and  supporting  information  systems.  Operations-oriented  data 
included  performance  definitions,  process  sequencing,  and  resource  requirements  data  along  with 
cost  Finally,  the  problems  and  improvement  related  ideas  firom  functional  experts  and  users 
addressed  a  wide  range  of  issues  such  as  the  identification  of  actual  problems,  symptoms  of 
problems,  personal  dislikes  about  the  process,  dupUcation  and  non-productive  aspects  within  the 
process,  process  adjustments,  technology  application,  and  process  control. 

The  data  collection  approach  was  initiated  with  the  development  of  a  stravraan  model,  in  the 
format  of  a  top-level  IDEFo  diagram.  This  strawman  model  provided  the  basis  to  understod  the 
scope  and  process  to  be  considered  during  the  ITI-ALC  analysis  effort,  and  provided  a  vehicle  for 
understanding  and  communicating  the  process  among  the  team  members. 

The  strawman  model  also  provided  the  foundation  on  which  to  develop  the  question  set  needed 
to  guide  the  data  collection  effort,  to  help  select  the  information  sites,  and  to  identify  the  specific 
information  sources.  The  strawman  model  also  provided  the  structure  around  which  the 
collected  data  was  documented  and  retrieved  during  model  development  effort.  The  data 
collection  effort  itself  was  accomplished  primarily  using  an  interview  approach,  and  supported  by 
SRA’s  Data  Collection  Device  (DCD).  Refer  to  Appendix  D  for  more  details  on  data  collection. 
The  collected  data  was  then  systematically  selected  fi-om  the  DCD,  analyzed,  and  used  to 
establish  the  "AS-IS"  models  which  represented  the  PDM  process  from  the  three  perspectives: 
functional,  data,  and  process  flow.  The  development  of  the  "AS-IS"  models  was  imtiated  first 
with  the  functional  model,  with  the  other  model  developments  initiated  shortly  thereafter,  and 
p,«int».iTiing  close  coordination  among  the  developments  to  ensure  the  traceability  of 
requirements  from  the  users  through  the  models. 

An  analysis  of  these  models,  along  with  the  user-identified  process  problems  and  improvement 
suggestions  produced  the  BPI  concepts,  which  in  turn  provided  the  basis  for  developmg  the 
corresponding  set  of  "TO-BE"  models.  As  with  the  "AS-IS"  model  development,  the  function^ 
model  provided  the  focal  point  for  all  the  “TO-BE”  model  development;  however,  all  the  model 
development  was  coordinated  to  ensure  completeness,  consistency,  and  traceability  of 
requirements. 

The  functional  and  processing  requirements  specified  via  the  "TO-BE’’ 
criteria  for  identifying  and  selecting  potential  technologies  for  implementing  the  TO-BE  PDM 
process.  Using  simulation  and  a  business  case  analysis,  a  benefit-versus-cost  trade-off  analysis 
determined  the  value-added  for  each  improvement  concept  and  the  selected  technologies. 

The  m-ALC  system  functional  requirements  were  derived  from  the  "TO-BE"  functional  model 
and  used  to  establish  the  SSS  for  the  ITI-ALC  system.  In  coordination  with  the  development  of 
the  SSS,  the  specification  for  the  ITI-ALC  system  was  represented  via  the  system  model  by  using 
information  from  the  benefit-versus-cost  trade-off  analysis,  the  functional  requirements  fron^e 
"TO-BE"  functional  model,  and  the  data  store  definitions  from  the  "TO-BE"  data  model.  This 
specification  identified  the  functionality  of  the  ITI-ALC  system,  the  data  stores  required,  the 
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interfaces  between  the  ITI-ALC  system  and  the  users,  the  internal  data  stores,  and  ^e  external 
data  systems.  Finally,  information  from  both  the  system  model  and  the  SSS  was  used  to  develop 
the  high-level  design  for  the  ITI-ALC  system  as  represented  in  the  SSDD. 


Although  there  are  many  methods  used  to  develop,  analyze  and  utilize  the  different  project 
entities  and  deliverables,  two  of  the  most  important  methods  for  any  Busmess  Process 
Reengineering  (BPR)  project  are  validation  and  traceability. 


2.1.1  Validation  Method 

User-focused  validation  of  all  the  models.  BPIs.  SSS  and  other  systems  engineering  efforts  were 
performed  during  their  development  to  ensure  their  respective  iirformation  was  user  onented  (see 
Figure  2-2).  The  "AS-IS"  models  were  the  first  models  to  be  validated. 


-AS-IS"  PM 


Figure  2-2.  Validation  is  User-Focused 

The  validation  of  the  "AS-IS"  functional  model  began  as  integral  part  of  its  development.  During 
the  strawman  development,  scenario-based  walkthroughs  were  held  with  function^  area  experts. 
During  each  data  collection  interview,  the  information  collected  from  previous  interviews  was 
verified  as  well  as  new  data  being  collected.  As  the  model  was  being  developed,  a  number  of 
walkthroughs  were  performed  and  readership  kits  were  distributed  among  the  team  tor 

model  review.  When  relatively  complete,  readership  kits  were  provided  to  the  ARINC 
area  experts  at  each  ALC  and  their  comments  were  used  to  refine  the  "AS-IS"  models.  ^ 

58  reviews  were  done  on  this  model  before  the  formal  walkthroughs  were  conducted  at  the 
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ALCs.  The  "AS-IS"  functional  model  was  then  presented  to  a  total  of  45  depot  maintenance 
personnel  as  part  of  the  formal  scenario  walkthrough  process. 

Since  the  development  of  the  "AS-IS"  data  model  was  based  on  the  information  specified  in  the 
"AS-IS"  functional  model,  much  of  the  validation  of  the  data  model  was^hieved  as  part  of  the 
validation  of  the  functional  model.  To  complete  the  validation  of  the  data  model,  readership  kits 
and  walkthroughs  were  used  with  functional  area  experts  fix)m  the  ITl-ALC  team. 

The  validations  of  the  "TO-BE"  functional  and  data  models  were  accomplished  in  conjunction 
with  the  validation  of  the  BPIs.  As  the  "TO-BE"  functional  and  data  models  were  develo^d, 
readership  kits  and  walkthroughs  were  used  with  functional  experts  firom  the  ITI-ALC  te^  ^e 
BPIs  identified  fi:om  the  analysis  of  the  "AS-IS"  models  and  the  development  of  the  TO-BE 
models  were  presented  to  the  Government  representatives,  fimctional  experts,  and  users 
representatives  to  ensure  the  accuracy  and  practicality  of  the  improvements  concepts  and  to 
expand  and  refine  the  BPIs. 

The  validation  of  "AS-IS"  and  "TO-BE"  process  and  simulation  models  was  accomplished,  to  a 
great  extent,  in  parallel.  As  segments  of  these  models  were  developed,  internal  walkthroughs 
were  held  to  ensure  tiie  process  flows  were  consistent  with  the  process  flow  understanding 
gained  during  the  development  of  the  functional  and  data  models.  Walkthroughs  of  &e  process 
flow  and  simulation  models  were  then  held  with  functional  experts  fi:om  the  ALCs  and  followed 
by  validation  trips  with  users  at  the  ALCs. 

The  SSS,  based  on  the  functional  requirements  derived  firom  the  "TO-BE"  functional  model,  w^ 
verified  by  members  of  the  ITI-ALC  team  using  reviews  at  various  pomts  dmng  its 
development.  When  completed,  a  validation  by  users  fi-om  the  ALCs  was  performed  usmg  a 
Joint  Application  Development  (JAD)  format.  During  this  two-day  workshop,  every  functional 
requirement  was  reviewed  by  the  users,  with  only  a  minimal  number  of  minor  changes  needed. 

2.1.2  Traceability  Method 

The  tracing  of  requirements  from  the  SSS  and  SSDD  back  to  the  user-specified  needs  ensures 
that  the  resulting  streamlined  depot  process  and  the  supporting  ITI-ALC  system  accurately  md 
completely  addressed  the  needs  of  the  users.  Requirements  traceability  may  be  viewed  as  a  quality 
assurance  method  used  to  ensure  the  following; 

•  Requirements  are  valid,  clear,  and  testable. 

•  Each  requirement  has  been  allocated. 

•  Each  step-in  design  is  capable  of  satisfying  all  the  allocated  requirements. 

•  No  requirements  have  been  inadvertently  omitted  or  lost. 

•  No  extraneous  requirements  are  introduced. 
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Requirements  traceability  also  provided  a  means  of  requirements  validation.  For  example,  a  low- 
level  requirement  should  be  traceable  to  a  higher-level  “parent”  requirement  based  on 
decomposition  of  the  parent  requirement.  If  it  is  not  a  pine  requirement,  part  of  a  set  of 
decomposed  requirements  or  a  pure  derived  requirement,  then  it  is  an  invalid  reqmrement  for  which 
there  will  be  no  traceabUity.  Traceability  includes  forward  tracing  during  development  forward 
from  the  user-specified  need  to  its  implementation  in  the  design,  and  backw^d  from  toe 
implementation  to  the  origin  of  toe  requirement  to  provide  context  for  any  requirement,  thus 
ensuring  a  clear  understanding  of  toe  intent  behind  toe  requirement 

Requirements  were  written  in  a  clear,  concise,  uniquely  identifiable  format  to  facilitate  traceability. 
Compound  requirements  were  avoided  unless  absolutely  necessary  and  only  when  those 
requirements  cannot  be  clearly  decomposed  for  allocation  to  finite  components  of  the  system 

design. 

Figure  2-3  depicts  the  traceability  scheme  use  for  toe  ITI-ALC  program  which  was  supported  hy  toe 
Requirements  Traceability  Component  (RTC).  The  RTC  tool  supports  multiple  users  to  perform 
requirements  analysis  by  presenting  and  organizing  data  in  a  way  that  establishes  and  mamrams 
relationships  between  multiple  sources  of  data.  The  RTC  tool  supports  the  development  of  these 
relationships,  or  links,  during  each  phase  of  the  development  cycle. 


TiaceabUily  exists  for  all  levels  of  the  "AS-IS"  models.  Implicit  traceability  to  Mgher-level  nodes 
of  the  model  md.  primary,  explicit  traceability  to  the  lowest-level  of  decomposition  of  each  model 
was  established.  For  the  m-ALC  “AS-IS”  models,  traceability  was  established  from  the  model’s 
lowest-level  nodes  to  die: 

•  Data  obtained  from  the  interviews. 

•  Data  regarding  other  systems  with  which  ITI-ALC  must  interface. 

•  Data  obtained  from  source  documents. 

•  Models  developed  for  the  IMIS  program. 

The  most  important  traceability  for  the  "TO-BE"  FM  was  to  the  "AS-IS"  FM.  Also,  because 
improvement  ideas  were  also  received  from  the  users,  related  models,  and  other  source  documents, 
there  was  also  traceability  from  the  "TO-BE"  FM  to  these  sources.  As  with  the  "AS-IS"  model 
tracing,  information  in  both  the  "TO-BE"  DM  and  "TO-BE"  PM  traced  back  to  the  "TO-BE"  FM. 

The  SSS  was  developed  to  contain  all  requirements  applicable  to  the  ITI-ALC  system  as  implied 
through  the  "TO-BE"  FM.  Therefore,  traceability  of  all  the  requirements  stated  in  the  SSS  were 

traced  back  to  the  "TO-BE"  FM. 

The  traceability  of  the  SSDD  was  to  both  the  System  Model  and  the  SSS.  The  System  Model 
established  the  high-level  design  of  the  ITI-ALC  system  in  terms  of  the  “system  processes”  and 
interface  requirements.  The  SSDD  used  the  structure  of  the  System  Model  to  specify  the 
components  of  the  ITT-ALC  system.  Therefore,  providing  direct  traceability  from  the  ITI-ALC 
SSDD  to  the  System  Model.  For  the  SSDD-to-SSS  traceability,  each  Hardware  Configuration  Item 
(HWCI),  CSCI,  and  manual  operation  of  the  SSDD  was  traced  to  the  system  fimctions, 
requirements,  and  interfaces  defined  in  the  SSS. 

2.1.3  Tool  Selection 

The  methodology  summarized  in  this  section  was  established  to  ensure  the  effective  development 
of  the  ITI-ALC  program.  Being  effective  required  that  each  step  builds  upon  the  results  of  the 
previous  steps  and  that  all  work  performed  had  a  direct  impact  on  the  completeness  and  quality 
of  the  final  product,  with  no  wasted  time  or  effort.  Furthermore,  although  less  important  than  the 
method  itself,  the  correct  tool  also  facilitated  effective  development. 

A  set  of  manual  and  automated  tools  facilitated  the  performance  of  each  step  in  the  methodology 
as  well  as  the  transfer  of  information  among  the  steps.  Table  2-1  identifies  the  selected  tools  the 
methodology  step  to  which  they  were  supplied,  and  the  producers  of  the  tools. 

The  data  collection  step  was  supported  by  the  DCD  tool.  The  DCD  was  implemented  on  a 
portable  computer  that  could  easily  be  taken  on  the  data  collection  trips  and  supported  the  use  of 
other  applications  such  as  the  IDEF  modeling  tools,  a  word  processor,  and  a  spreadsheet.  The 


DCD  was  designed  with  the  structures  necessary  to  capture  all  the  data  that  might  be  identified 
during  the  data  collection  efforts,  and  with  a  user  interface  that  allowed  for  the  input  and  output 
of  information  in  a  manner  intuitive  to  the  interviewers  and  modelers. 


Table  2-1.  Tools  Used  to  Implement  the  Methodology 


Methodology  Step 

Tool 

Producer 

Data  Collection 

Data  Collection  Device  (DCD) 

SRA  Corp. 

"AS-IS"  and  "TO-BE"  Functional 

Modeling 

Design/IDEF,  Version  3.5 

Meta  Software 

"AS-IS"  Data  Modeling 

Design/IDEF,  Version  3.5 

Meta  Software 

"TO-BE"  Data  Modeling 

ERWin,  Version  2.1 

Logic  Works 

"AS-IS"  and  "TO-BE"  Process  Modeling 

ProSim,  Version  2.0 

KBSI,Inc. 

"AS-IS"  and  "TO-BE"  Simulation 

WITNESS,  Version  6.0 

AT&T 

Site  Selection 

Expert  Choice  (8,0) 

Expert  Choice  Inc. 

FEA 

TurboBPR2.0 

SRA  Corp. 

SW  Cost  Estimates 

Checkpoint  (2.1)  . 

SPR 

System  Model 

System  Architecture,  Version  3.0  G6A 

Popkin  Software 
Systems 

Traceability 

The  Requirements  Traceability  Component 
(RTC)  of  WISARD,  Version  2.2 

SRA  Corp. 

Documentation 

Microsoft  Word,  Version  6.0  and  other 
Microsoft  Office  tools 

Microsoft 

The  functional  models  and  the  "AS-IS"  data  model  were  supported  by  the  Design/IDEF.  The 
capabilities  of  the  functional  modeling  aspect  of  Design/IDEF  were  well  known  to  the  ITI-ALC 
team  and  the  capability  to  effectively  transfer  information  from  the  functional  model  to  the  tools 
supporting  other  steps  within  the  methodology  had  already  been  established  and  proven.  The 
selection  of  Design/IDEF  to  support  the  development  of  the  "AS-IS"  data  model  was  also  based 
on  past  experience  with  the  tool  and  the  knowledge  that  the  last  release  of  the  tool  had 
significantly  improved  capabilities. 

While  the  capabilities  of  Design/IDEF  to  support  the  "AS-IS"  data  modeling  were  adequate,  an 
analysis  of  the  ERWin  tool  demonstrated  that  it  had  key  benefits  over  the  data  modeling  aspects 
of  Design/IDEF.  Specifically,  ERWin  provided  improved  transfer  of  attributes  among  the 
entities  to  better  support  the  implementation  of  the  IDEFix  notation.  Therefore,  ERWin  was 
selected  for  use  in  developing  the  "TO-BE"  data  model. 

ProSim  was  selected  to  implement  the  process  flow  model  because  it  was  the  only  commercially 
available  tool  advertised  to  support  the  IDEF3  notation.  Similarly,  WITNESS  was  selected  to 
develop  the  simulation  because  an  automated  transfer  of  information  from  ProSim  to  WITNESS 
that  facilitated  the  development  of  a  simulation  model  directly  from  the  process  flow  model  was 
advertised  to  exist  and  fully  operational. 
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System  Architecture  was  the  tool  selected  to  support  the  development  of  the  system  model.  T^s 
tool  supported  a  top-down  approach  to  modeling  that  facilitated  the  traceability  between  the  TO- 
BE"  functional  and  system  models,  and  provided  the  capability  to  represent  information  stores 
and  interfaces  within  its  notation.  These  capabiUties  provided  an  effective  means  of  representing 
the  ITI-ALC  system. 

Expert  Choice  (EC),  TurboBPR,  and  Checkpoint  were  application  tools  used  in  some  aspect  of 
the  development  of  the  ITI-ALC  Business  Case.  Expert  Choice  was  used  to  select  ihe  sites 
featured  in  the  business  case.  TurboBPR  was  used  to  perform  the  Functional  Economic  Analysis 
(FEA)  and  Checkpoint  is  a  system/project  cost  estimation  tool  used  to  determme  some  of  the 
costs  with  the  ITI-ALC  system  implementantion. 

The  RTC  tool  was  developed  by  SRA  to  support  the  tracing  of  requirements  throughout  the  ITI- 
ALC  program  to  ensure  the  user  needs  were  completely  and  accurately  represented  in  the  SSS 
and  SSDD.  This  development  was  necessary  because  no  COTS  tools  existed  nor  did  the 
capability  exists  within  the  other  tools  selected  for  this  project. 

Microsoft  Word  was  used  to  document  all  the  results  of  the  ITI-ALC  program.  This  appli^tion 
was  selected  because  it  accepted  the  electronic  transfer  of  information  from  all  the  other  selected 
tools  and  facilitated  the  formatting  desired  for  the  various  reports.  Other  apphcaUons  used 
throughout  the  program  were  Microsoft  Excel  and  Microsoft  PowerPoint.  Refer  to  Appendix  A 
for  details  on  how  the  documentation  was  accomplished. 


3.  DISCUSSION  AND  RESULTS 


The  products  of  this  project  were  a  set  of  artifacts  and/or  deliverables  that  documented  the 
knowledge  gained  during  the  performance  of  the  ITI-ALC  Phase  I  program.  Specifically,  the 
products  produced  were: 

IDEFo  Functional  Models  (FMs) 

“AS-IS”  -  Represents  the  current  depot  maintenance  activities. 

“TO-BE”  -  Represents  the  recommended  or  future  depot  maintenance  activities. 

IDEFix  Data  Models  (DMs)  * 

“AS-IS”  -  Represents  the  logical  information  struchire  that  supports  the  current  depot 

maintenance  process. 

“TO-BE”  -  Represents  the  recommended  logical  information  structure  to  support  the 
future  depot  maintenance  process. 

IDEF3  Process  Models  (PM)  &  WITNESS  Simulations 

“AS-IS”  -  Represents  the  dynamic  aspects  of  the  current  depot  maintenance  activities 

as  defined  by  the  “AS-IS”  FM. 

“TO-BE”  -  Represents  the  dynamic  aspects  of  the  recommended  or  future  depot 
maintenance  activities  as  defined  by  the  “TO-BE  FM. 


System  Model  (SM) 

—  Represents  the  ITI-ALC  system. 

Business  Case  (BC)  , 

-  Documents  the  costAienefit  analysis  for  the  “TO-BE”  implementation. 


System/Segment  Specification  (SSS) 

-  Documents  the  requirements  for  the  retool  needed  m  the 


"TO-BE’^ 


implementation. 


System/Segment  Design  Document  (SSDD) 

-  High-level  design  for  the  “TO-BE”  implementation. 


IMIS/ITI-ALC  System  Comparison  j 

-  Functional  comparison  between  the  IMIS  and  ITI-ALC  system,  and  a 
demonstration  plan  for  the  next  phase  of  ITI-ALC. 


Demonstration  Plan  a  t 

-  Planning  document  for  a  Phase  11  demonstration  of  the  ITI-ALC  system. 


The  following  sections  present  discussions  pertaining  to  each  of  the  artifacts/deliverables 
developed  for  this  effort  as  well  as  possible  recommendations  and/or  specific  lessons  learned 
during  its  development.  This  list  of  possible  recommendations  and/or  lessons  learned  is  also 
repeated  in  Appendix  C,  organized  by  artifact/deliverable. 
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3.1  ”AS-IS"  MODELS 

A  set  of  “AS-IS”  models  were  developed  to  provide  a  complete  and  accmate  representation  of 
the  correct  PDM  process.  These  model  developments  were  based  on  information  collected  from 
each  of  the  five  ALCs  by  using  a  systemactic  data  collection  approach.  This  data  collection 
approach  is  documented  in  Appendix  D. 

Using  the  collected  information,  die  functional  model  development  was  initiated  first  and 
followed  by  the  data  model  development.  The  development  of  the  “AS-IS”  process  model  was 
initiated  later  in  the  program,  following  the  addition  of  that  requirement  to  the  ITI-ALC  program. 

3.1.1  ”AS-IS*'  Functional  Model 

The  “AS-IS”  FM  is  a  representation  of  the  existing  Air  Force’s  depot  maintenance  process.  The 
purpose  of  this  model  is  to  provide  a  means  of  logically  illustrating  and  documenting  the 
functions  currently  performed  within  the  depot  maintenance  process  along  with  the  informational 
relationships  among  the  functions.  The  primary  goal  of  the  "AS-IS"  FM  was  to  aggregate  the 
process  description  data  collected  from  the  five  ALCs  into  a  single  representation  of  a  depot 
maintenance  process  that  highlights  duplications  and  similarities  among  Ae  ^^Cs  ^  well  ^ 
difference  among  the  ALCs.  The  scope  of  the  ITI-ALC  program,  and  therefore  &e  AS-IS  FM, 
is  primarily  PDM,  with  a  limited  look  at  components  and  backshop.  The  viewpoint  of  this  model 
is  that  of  the  maintenance  personnel. 

3.1.1.1  ’’AS-IS"  FM  Development 

The  strawman  model  provided  the  starting  point  for  the  development  of  the  "AS-IS"  FM.  Using 
the  data  collected  during  the  data  collection  trips,  the  strawman  model  was  refined  and  expanded 
into  the  "AS-IS"  FM. 

The  first  step  in  developing  the  "AS-IS"  FM  was  to  refine  the  purpose  and  vie\\i)oint  of  ±e 
strawman  model  to  formalize  the  scope  and  content  of  the  information  to  be  mcluded  m  the 
model.  Establishing  an  effective  purpose  and  viewpoint  for  the  "AS-IS"  FM  was  very  important 
because  it  also  established  the  scope  and  viewpoint  for  all  subsequent  models. 

Using  the  strawman  model  as  a  reference  point,  data  searches  were  performed  through  the  DCD 
to  systematically  access  and  organize  information.  This  information  was  analyzed  to  support  the 
decomposition  of  each  activity  within  the  strawman  and  resulted  in  a  complete  and  accurate 
description  of  the  depot  maintenance  process  as  contained  within  the  scope  of  the  program. 

This  data  retrieval  and  model  development  process  also  identified  holes  and  inconsistencies  in 
the  data  contained  in  the  DCD.  In  order  to  correct  these  infoimation  problems,  one  or  more  of 
the  following  actions  were  taken. 

1.  The  original  data  collected  (e.g.,  notes,  tapes,  etc.)  from  the  interview(s)  was  reviewed 
and  compared  to  the  data  contained  within  the  DCD. 

2.  Discussions  were  held  with  flmctional  experts  available  on  the  ITI-ALC  team. 
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3.  Each  ALC  had  an  ITI-ALC  point-of-contact  (POC)  on-site  who  researched  or  collected 
more  data  to  resolve  information  problems. 


At  intervals  during  the  model  development,  scenarios  were  developed  and  used  to  present  the 
model  to  members  of  the  ITI-ALC  team.  These  walkthroughs  provided  two  benefits.  First,  they 
provided  a  communication  vehicle  to  ensure  that  the  collected  information  was  being  interpreted 
correctly;  therefore,  ensuring  that  the  model  was  validated  within  the  ITI-ALC  team.  Second,  the 
walkthroughs  provided  a  means  by  which  the  ITI-ALC  team  could  understand  and  discuss  Ae 
depot  maintenance  process.  This  foimdation  of  understanding  was  critical  for  fotoe  discussion 
with  depot  maintenance  personnel  and  for  identifying  and  evaluating  process  improvements. 
This  process  of  data  accessing,  analysis,  modeling,  and  validation  continued  until  the  model  was 
to  the  level  judged  necessary  to  satisfy  the  requirements  of  the  ITI-ALC  program. 


Along  with  each  diagram  was  developed  a  narrative  was  described,  at  a  minimum,  the  primary 
process  flow  through  the  diagram.  Additional  process  flow  information  was  included  as 
appropriate,  but  the  overall  narrative  was  limited  to  three-fourths  of  a  page  due  to  the  page-pair 
format  used  to  document  the  model.  The  joint  development  of  the  diagram  and  the  narrative  also 
provided  a  means  of  verifying  the  completeness  of  information  being  presented  in  the  diagram. 

As  each  arrow  was  placed  on  a  diagram  and  identified,  a  definition  was  associated  with  the 
arrow.  The  arrow  names  and  their  definitions  were  then  pulled  from  the  model,  along  vnth  a 
reference  indicating  the  diagram  on  which  the  arrow  appeared,  to  form  the  glossary  for  the  "AS- 
IS"  FM,  which  was  merged  with  the  "AS-IS"  DM  glossary. 

3.1.1.2  “AS-IS”  FM  Validation 

The  review  and  validation  of  the  “AS-IS”  FM  was  accomplished  as  an  integral  part  of  the  model 
development  effort.  The  review  and  validation  approach  was  different  at  the  various  stages  of 
the  model  development;  however,  the  underlying  purpose  for  validation  was  to  ensure  the 
model’s  accuracy  and  completeness,  and  to  gain  user  involvement  in  the  models  and  thereby  gain 
model  acceptance  and  ownership  by  the  users.  Following  each  review  and  validation,  all 
comments  received  were  documented  and  analyzed  for  incorporation  into  the  model. 

Throughout  the  development  of  the  "AS-IS"  FM,  internal  walkthroughs  were  held  among 
members  of  the  ITI-ALC  team,  which  included  fimctional  experts.  These  walkthroughs  were 
based  on  scenarios  which  overlaid  onto  the  fimctional  model.  Also,  these  walkthroughs  ensured 
that  information  obtained  by  the  team  was  included  in  the  model  and  provided  a  common  process 
understanding  among  the  team  members.  This  common  understanding  was  required  for  the 
identification  and  discussions  of  the  Business  Process  Improvements  (BPIs). 

As  the  model  neared  completion,  14  interview  kits  were  provided  to  members  of  the  ITI-ALC 
team.  This  cycle  was  followed  by  44  interview  kits  being  sent  to  ARINC  fimctional  experts  and 
depot  maintenance  personnel  located  at  the  ALCs.  Finally,  six  scenario-based  walkthroughs 
were  developed  and  presented  to  45  users  from  four  ALCs.  The  model  validations  were 
performed  using  a  method  called  the  “IDEF  schematic.”  In  IDEF,  a  schematic  is  all  of  Ae 
lowest-level  IDEF  diagrams  connected  together  to  form  a  complete  picture  of  the  activity  being 
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modeled.  The  positive  reaction  received  from  these  walkthroughs  indicated  the  "AS-IS"  FM 
provided  a  complete  and  accurate  representation  of  the  PDM  portion  of  depot  maintenance. 

3.1.13  “AS-IS”  FM  Traceability 

The  "AS-IS"  FM  is  developed  to  represent  an  accurate  and  complete  description  of  the  current 
depot  maintenance  process,  witii  a  focus  on  PDM.  Since  the  information  used  to  develop  the 
"AS-IS"  FM  came  from  the  users  at  the  various  depot  sites,  it  was  necessary  to  trace  information 
in  the  "AS-IS"  FM  back  to  the  users  to  ensure  that  all  the  user  needs  were  incorporated  in  the 
model. 

This  traceability  was  accomplished  by  tracing  each  activity  in  the  "AS-IS"  FM  to  an  interview 
number,  and  by  ensuring  that  every  activity  documented  in  the  DCD  was  represented  in  the  "AS- 
IS"  FM. 


3.1.1.4  "AS-IS”  FM  Possible  Recommendations  and  Lessons  Learned 

The  following  lessons  were  learned  during  the  data  collection,  and  the  development  and  use  of 

the  "AS-IS"  FM.  Possible  recommendations  may  also  apply. 

•  Significant  preparation  for  the  data  collection  trips  was  critical. 

User  acceptance  is  critical  to  any  system  development  effort.  Because  the  data  collection 
trips  were  the  first  technical  interface  with  the  users  and  because  humans  have  a  tendency  to 
let  "first  impressions  be  lasting  impressions",  the  data  collection  trips  provided  the  first  and 
best  opportunity  to  gain  the  users'  acceptance  of  and  involvement  in  the  ITI-ALC  program. 
To  acquire  the  desired  positive  impression,  the  data  collection  trips  must  be  well  planned  and 
effectively  implemented. 

The  data  collection  for  the  ITI-ALC  program  was  effective.  The  strawman  model  provided  a 
solid  baseline  for  identifying  the  data  to  be  collected,  identifying  the  sources  for  the  data, 
providing  a  good  vehicle  for  training  the  interviewees  about  the  process,  and  understanding 
where  each  interviewee  fit  into  the  depot  maintenance  process  prior  to  starting  an  interview. 

The  question  set  provided  an  effective  means  for  supporting  the  data  collection  trips.  While 
used  only  to  a  limited  extent  during  the  actual  interviews,  the  primary  value  of  the  question 
set  was  in  training  the  interviewers  prior  to  the  data  collection  trips.  Much  like  the  notes 
developed  and  used  by  an  orator  to  prepare  and  practice  the  speech  but  used  only  as  a  quick 
reference  during  the  delivery  of  the  speech,  the  primary  value  of  the  question  set  was  received 
during  the  preparation  of  the  data  collection  effort. 
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•  Maintain  a  list  of  documents  collected,  user  identified  problems  and  improvement  ideas, 
and  a  list  of  activities  already  defined. 

To  implement  an  effective  and  efficient  interview,  each  interview  should  confirm  as  well  as 
build  upon  information  jfrom  previous  information.  To  support  this  information  building 
process,  an  up-to-date  list  of  key  information  should  be  maintained  as  reference  and  to 
reduce  uimecessary  effort. 

A  valuable  source  of  information  are  documents,  forms,  etc.  that  are  used  within  the  process 
being  described.  Listing  these  documents,  along  with  a  short  description  of  each  provides  an 
effective  way  of  understanding  what  the  interviewee  is  meamng  when  he  indicates  one  of 
these  documents  and  also  provides  a  way  of  tracking  which  documents  have  already  been 
requested  or  received. 

In  a  similar  maimer,  maintaining  a  list  of  process  problems  and  solutions  provided  during 
previous  intervals  provided  a  guide  to  confirm  the  same  ideas  from  multiple  sources  and  to 
build  upon  the  ideas  presented. 

A  list  of  activities  performed  within  the  process  was  also  maintained  in  the  DCD  as  a  point  of 
reference.  This  list  provides  a  means  of  learning  the  terminology,  of  mapping  previous 
interview  information  with  information  currently  being  collected,  and  identifying  similar 
functions  identified  by  different  terminology.  Using  this  approach  produces  a  controlled  but 
not  restricted  list  of  standardized  activity  identifiers. 

•  The  data  collection  interview  must  be  carefully  controlled  by  the  interviewer. 

Interview  control  must  be  balanced  between  strict  control  where  the  interviewee  is  answering 
very  specific  questions,  and  loose  control  where  the  interviewee  is  doing  all  ffie  talking. 
Asking  questions  based  very  tightly  on  the  question  set  tended  to  limit  the  information 
provided  by  the  interviewee  and  assumed  that  the  interviewer's  prior  understanding  of  the 
process  was  accurate  because  the  interviewer  was  directing  the  information  being  obtained. 
Initiating  the  interview  with  "tell  me  what  you  do"  does  not  focus  the  interviewee  so  they 
tend  to  ramble,  hoping  they  say  something  that  is  of  importance  to  the  interviewer. 

The  most  effective  approach  was  to  explain  our  current  understanding  of  the  depot 
maintenance  process,  describe  the  t5q)®s  of  information  desired,  and  then  ask  questions  that 
focused  the  interviewees  discussion  aroimd  the  information  needed.  It  was  the  interviewers 
responsibility  to  evaluate  the  usefulness  of  the  information  being  provided  and  determine 
when  additional  questions  were  needed  to  refocus  the  discussion  to  ensure  that  the  data 
requirement  of  the  question  set  were  satisfied.  By  allowing  the  interviewee  to  describe  the 
process  in  their  terms  reduced  their  need  to  adjust  to  an  unfamiliar  way  of  discussing  their 
process.  As  a  general  rale  of  thumb,  it  was  effective  at  2  to  3  minute  intervals  to  ask  a 
refocusing  question  or  provide  some  type  of  indication  to  the  interviewee  that  appropriate 
information  was  being  provided. 
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Scheduling  the  interview  for  the  one-hour  duration  was  sufficient  to  collect  signific^t 
information  and  not  lose  the  attention  of  the  interviewee.  To  aid  in  accurately  docmnenting 
the  collected  information,  a  one-hour  period  should  also  be  schediiled  immediately  following 
the  interview  to  formally  record  the  information  while  it  is  still  fresh. 

•  An  end-of-day  meeting  should  be  held  during  data  collection  trips. 

The  data  collection  teams  should  meet  at  the  end  of  each  day  to  review  the  events  of  the  day 
and  to  distribute  information  among  the  team  members.  This  review  should  address  the 
individuals  interviewed  during  the  day,  a  summarization  of  the  information  obtained, 
identification  of  missing  mfonnation,  and  a  review  of  the  reference  list  contaimng  the 
documents  collected,  systems  and  activities  identified,  and  process  problems  and 
improvement  ideas.  The  schedule  for  the  next  set  of  interviews  should  also  be  reviewed  and 
coordinated  to  ensure  that  each  interview  will  be  as  productive  as  possible. 

•  Allowing  analysis  time  between  data  collection  trips  facilitates  the  collection  of  accurate 
and  complete  data. 

Obtaining  accxirate  and  complete  data  requires  that  the  interviewee  and  interviewer  have  the 
same  interpretation  of  the  questions  being  asked  during  data  collection,  that  the  interviewee  is 
providing  accurate  information,  and  that  the  interviewer  is  interpreting  the  answers  correctly. 

The  best  way  to  identify  discrepancies  in  the  collected  data  is  to  obtain  and  analyze 
information  related  to  the  same  aspect  of  the  process  but  from  different  sources.  This 
analysis  is  facilitated  by  allowing  sufficient  time  between  the  interview  trips.  This  analysis 
verifies  the  process  understanding  by  the  interviewers,  verifies  consistency  among  the  data, 
and  identifies  holes  in  the  data. 

•  Effective  scoping  of  the  functional  model  is  important  to  maximize  the  benefits  received 
from  a  BPR  effort. 

The  focal  point  of  the  ITI-ALC  program  was  the  PDM  mechanics  within  the  Air  Force's  Air 
Logistics  Centers.  However,  the  scope  was  expanded  at  program  initiation  to  include  within 
the  scope  of  ITI-ALC's  BPR  analysis  the  operational  environments  impacting  the  mechanic's 
performance  effectiveness.  This  expanded  scope  provided  the  foundation  needed  to 
effectively  streamline  the  mechanic  process  since  much  of  the  work  perforrned  by  the 
mechanic  was  a  duplication  or  continuation  of  the  work  performed  by  other  individuals. 
Therefore,  restricting  the  scope  of  die  ITI-ALC  program  just  to  the  mechanic  would  have 
limit  the  BPIs  identified  for  the  mechanic  and  would  have  reduced  improved  effectiveness 
provided  by  the  proposed  ITI-ALC  system. 
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3.1.2  ”AS-IS”  Data  Model 

The  "AS-IS"  DM  is  a  logical  representation  for  how  the  data  used  within  the  current  depot 
maintenance  process  is  organized  and  related.  The  purpose  of  this  model  is  to  provide  a  tool  for 
analyzing  the  effectiveness  of  the  logical  organization.  The  "AS-IS"  DM  representation 
identifies  groupings  of  data  called  entities  and  properties  of  those  entities  called  attnbutes. 

3.1.2.1  "AS-IS"  DM  Development 

The  data  for  the  "AS-IS"  DM  model  was  obtained  from  a  number  of  sources,  but  the  primary 
data  source  was  the  arrows  from  the  “AS-IS”  FM.  The  arrows,  representing  inputs,  outputs,  and 
controls,  fi’om  the  lowest-level  nodes  of  the  "AS-IS"  FM  were  selected  and  separated  into  three 
categories:  (1)  activation  triggers,  (2)  physical  items,  and  (3)  pure  data. 

The  physical  items  are  arrows  that  represent  actual  tangible  items  such  as  an  aircraft  or  an  engine 
that  moves  between  activity  nodes.  These  arrows  had  to  be  examined  to  determine  what  data 
about  these  physical  items  actually  flows  through  the  system.  In  theory,  it  is  possible  that  no  data 
may  be  associated  with  these  arrows,  but  in  practice  we  found  that  data  was  maintained  about 
each  of  the  physical  data  arrows  somewhere  within  the  system.  This  data  was  added  to  the  data 
represented  by  the  data  items.  In  many  cases,  these  data  elements  represented  ag^egates  of 
separate  facts  (properties);  therefore,  they  were  further  decomposed.  The  individiM  facts 
became  the  attributes.  The  attributes  were  grouped  into  logical  groupings  to  form  the  entities. 

The  development  of  the  “AS-IS”  DM  was  initiated  using  information  represented  by  the  arrows 
on  the  strawman  model.  Then  at  each  point  during  the  model  development  effort  when  the  “AS- 
IS”  FM  was  considered  baseline,  the  arrows  were  collected  via  the  arrow  report  from  the  “AS- 
IS”  FM  and  used  to  verify,  expand,  and  refine  the  “AS-IS”  DM. 

The  listing  of  arrows  was  grouped  into  sets  of  like  elements.  From  these  sets,  potential  entities 
and  attributes  were  identified,  and  some  of  the  relationships  among  entities  specified,  and 
additional  entities  were  suggested.  Using  the  notation  of  IDEFix,  this  process  was  aided  by 
trying  sample  instance  tables  on  the  potential  entities  and  relationships  to  determine  whether  the 
modeled  business  were  reasonable.  The  attnbutes  were  then  added  to  further  define  the  entities. 
As  the  DM  was  being  developed,  information  voids  and  questions  were  related  back  to  the  FM  to 
help  enhance  the  completeness  and  accuracy  of  the  FM. 

Normalization  rules  were  then  applied.  The  many-to-many  relationships  were  resolved  through 
the  development  of  associative  entities.  However,  it  was  decided  that  recursive  relationships 
would  be  allowed  unless  additional  attnbutes  needed  to  be  maintained  in  a  resolving  associative 
entity  (i.e.,  the  model  would  not  attempt  5th  normal  form).  We  also  decided  that  we  would  allow 
for  attributes  having  null  values  (i.e.,  the  model  would  not  strive  for  4th  normal  form).  We  did 
not  see  any  value  added  by  introducing  the  additional  model  complexity  that  the  many  resolving 
category  entities  would  produce. 
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A  decision  had  to  be  made  regarding  the  handling  of  the  many  “non-atomic”  data  elements  Aat 
were  identified.  An  atomic  (normalized)  data  element  carries  only  one  piece  of  information. 
However,  in  the  “AS-IS”  world,  many  common  data  elements  contain  multiple  pieces  of 
information.  For  example,  the  well-used  document  number  contains  the  Stock  Record  Account 
Number  (SRAM),  the  document  date,  and  a  serial  number.  The  SRAM  itself  coritains  multiple 
types  of  data.  This  could  be  broken  into  numerous  atomic  attributes.  However,  it  was  decided 
that  to  do  so  in  the  “AS-IS”  DM  would  make  it  more  difficult  for  fimctional  experts  to  recognize 
the  data  and  be  able  to  validate  the  model.  Instead,  it  was  decided  to  leave  these  types  of  non- 
atomic  attributes  in  the  model,  but  to  document  the  elements  of  information  within  them  m  the 
glossary.  This  allowed  for  normalization  in  the  “TO-BE”  DM. 

If  the  “AS-IS”  FM  was  decomposed  to  its  lowest  possible  level  and  involved  no  information 
bundling,  each  data  arrow  would  represent  a  single  data  element  that  would  translate  into  an 
attribute.’  Since  this  level  of  decomposition  was  not  necessary  and  too  time  consuming  for  the 
analysis  for  which  the  model  was  intended,  the  data  bundling  necessitated  access  to  additional 
data  sources  through  which  the  individual  data  elements  could  be  identified.  These  data  sources 
were  identified  by  the  mechanisms  on  the  “AS-IS”  FM.  These  sources  included  mdiVK^s 
performing  the  maintenance  process,  documentation  describing  the  process,  artifacts  from 
existing  AIS,  and  forms  on  which  carried  the  information  through  the  depot  maintenance  process. 
This  process  was  augmented  by  using  functional  experts  to  explain  and  elaborate  the  data  foimd. 

To  maximize  the  coordination  of  the  “AS-IS”  DM  with  the  FM  as  well  as  other  program  efforts, 
the  wording  and  the  data  element  descriptions  or  definitions  firom  the  DM  glossary  were 
coordinated  with  the  FM.  In  addition.  The  Department  of  Defense  (DoD)  Enterprise  Model 
Volume  I:  Strategy  Activity  and  Data  Models  was  referenced  to  maximize  the  standardization  of 
terms  and  definitions  with  other  programs.  JLSC’s  IFB  IDEFu  Model  and  Glossary  (key-based) 
Technical  Report,  28  February  1994  was  also  referenced  to  ensure  linkage  with  the  bigger  picture 
of  depot  maintenance  it  provides. 

3.1.2.2  "AS-IS"  DM  Validation 

The  DM  validation  process  was  accomplished  through  walkthroughs  with  function^  area 
experts,  author-readership  cycles,  and  validation  trip  walkthroughs.  The  vali^tion  adihessed 
two  perspectives.  First,  the  reviews  ensured  that  correlation  existed  between  the  FM  and  DM, 
the  DM  conformed  to  the  syntax  of  the  modeling  language,  all  terms  were  fully  defmed  m  the 
glossary,  and  the  correlation  with  other  depot  maintenance  information  was  maximized.  Second 
the  validation  process  ensured  the  model  provided  complete,  accurate,  and  understandable  logical 
representation  of  the  depot  maintenance  current  information.  After  each  review,  the  DM  was 
updated  to  reflect  the  comments  received. 

Because  the  “AS-IS”  FM  controlled  the  baseline  information  contaiiied  in  the  “AS-IS”  DM,  the 
validation  of  the  FM  provided  a  significant  step  toward  the  validation  of  the  DM.  Additional 
validation  of  the  DM  was  accomplished  through  walkthroughs  with  functional  area  experts, 
author-reader  cycles,  and  validation  trip  walkthroughs. 
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It  was  soon  evident  that  presenting  the  model  as  a  whole  did  not  facilitate  effective  review  of  the 
model.  While  the  FM  is  segmented  into  easily  understandable  diagrams  by  its  nature  as  a 
hierarchical  model,  the  DM  does  not  have  this  feature.  Therefore,  the  DM  was  segmented  into 
broad  classes  of  information  types  called  subject  areas.  The  attributes  in  each  entity  were 
examined  to  evaluate  what  type  of  information  was  represented.  Then  entities  with  similar  types 
of  information  were  grouped.  By  examining  the  classes  of  information  represented,  seven  major 
subject  areas  emerged:  MATERIEL,  PLANS,  ACTIONS,  FACILITIES,  LOCATION, 
ORGANIZATIONS,  and  PEOPLE.  These  correspond  to  seven  of  the  13  strategic  level  data 
“buckets”  in  the  DoD  Enterprise  Data  Model. 

The  "AS-IS"  DM  was  presented  in  both  presentation  and  formal  documentation  using  a  page-pair 
format  based  on  the  subject  areas.  The  narrative  for  the  subject  area  was  on  the  top-page  and  the 
subject  area  DM  on  the  bottom  page.  Included  with  the  page-pair  model  was  a  cross-reference 
chart  mapping  each  entity  to  the  subject  area  and  the  glossary  containing  definitions  of  all 
entities  and  attributes  in  the  model. 

3.1.2.3  "AS-IS"  DM  Traceability 

The  traceability  of  the  "AS-IS"  DM  was  accomplished  by  mapping  the  entities  of  the  DM  to  the 
data  arrows  of  the  "AS-IS"  FM.  Because  of  the  bundling  of  data  in  the  FM  arrows,  mapping  to 
the  attribute  level  of  the  DM  was  determined  to  be  impractical.  When  an  arrow  mapped  to  an 
attribute  that  migrated  to  several  entities,  such  as  a  foreign  key,  it  was  mapped  only  to  the  entity 
that  owned  the  attribute. 


The  results  of  the  mapping  was  presented  as  a  matrix  so  that  the  DM-to-FM  and  the  FM-to-DM 
could  both  be  readily  seen.  This  mapping  was  documented  within  the  RTC. 

3.1.2.4  “AS-IS”  DM  Possible  Recommendations  and  Lessons  Learned 

The  following  lessons  were  learned  during  the  development  and  use  of  the  "AS-IS"  DM  and/or 

may  present  possible  recommendations. 

•  Developing  the  DM  in  conjunction  with  the  FM  resulted  in  a  tight  linkage  between  the 
models. 

The  original  plan  expected  that  modeling  the  functions  performed  would  vmcover  many  of  the 
data  requirements.  However,  we  formd  that  the  data  modeling  also  imcovered  additional 
functional  requirements.  This  two-way  interchange  could  take  place  by  walking  through 
scenarios  in  the  FM  and  identifying  the  information  within  the  DM  that  supports  the 
scenarios.  Because  of  the  tight  linkage  between  the  DM  and  the  FM,  the  merging  of  the  two 
glossaries  was  feasible  and  more  useful  than  the  presentation  of  individual  glossaries. 
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The  use  of  Design/IDEF  as  the  IDEFix  modeling  tool  required  some  other  conventions 
be  adopted. 

DoD  8320.1-M-l  mandates  that  a  data  element  name  consists  of  a  prime  word  name  with  its 
modifiers  and  the  generic  element  name  with  its  modifiers.  This  convention  was  not  adopted 
for  two  reasons.  The  tool  automatically  draws  the  entity  boxes  large  enough  to  accommodate 
the  largest  entity  or  attribute  name.  Using  fulling  qualified  names  made  the  diagram  much 
too  crowded.  Even  using  only  the  prime  word  name  had  to  be  relaxed  because  the  tool 
requires  that  each  attribute  within  the  model  be  unique.  So  in  some  cases,  the  attribute  name 
was  concatenated  on  the  entity  name  to  form  a  unique  attribute  name.  For  example,  more 
than  one  entity  has  an  identifier  attribute.  To  keep  them  unique,  they  became  facility- 
identifier,  cost-center-identifier,  etc. 

The  data  models  could  not  be  reviewed  effectively  as  a  whole. 

The  data  models  representing  the  "AS-IS"  and  "TO-BE"  depot  maintenance  process  were  too 
large  and  complex  to  be  effectively  presented  and  reviewed  as  a  single,  complete  model.  To 
facilitate  the  reviews  and  and  model  discussions,  the  model  was  divided  into  subject  areas, 
with  each  subject  area  containing  groupings  of  related  infomation.  These  subject  areas  were 
then  same  enough  to  support  and  effective  and  focused  review.  In  addition,  the  subject  areas 
also  allowed  for  the  use  of  an  effective  model  documenation  format.  This  format  was  a  page- 
pair  structure  consisting  of  a  subject  area  diagram  and  the  associated  text  describing  the 

diagram. 

The  normality  rules  had  to  be  applied  in  a  manner  that  maximized  the  development 
and  usefulness  of  the  "AS-IS"  DM. 

A  primary  objective  of  data  modeling  is  to  describe  system  data  to  the  level  where  the  data  is 
completely  normalized  at  the  atomic  level  without  duplication  of  null  attribute  values.  While 
this  is  an  optimal  goal  to  strive  for,  a  trade-off  must  sometimes  be  made  between  the  optimal 
data  model  and  a  useful  data  model.  For  the  ITI-ALC  program,  this  optimized  trade-off  was 
defined  by  the  following  conditions. 

1  _  The  model  would  be  developed  to  the  3rd  normal  form.  The  5th  normal  form  was  not 
implemented  since  recursive  relationships  would  be  allowed  unless  additional 
attributes  needed  to  be  maintained  in  a  resolving  associative  entity.  The  4th  normal 
form  was  not  implemented  to  allow  for  the  use  of  nuU  atrtbute  values.  This  avoided 
the  creation  of  many  category  entities  needed  to  eliminate  null  attributes  but 
decreased  the  ease  of  understanding  the  data  contained  in  the  model. 

2.  Not  all  data  was  broken  down  to  its  atomic  or  normalized  data  elements.  Trying  to 
achieve  all  the  atomic  data  elements  involved  significant  effort  but  provided  minimal 
benefit  return.  Also,  representing  the  atomic  data  elements  would  significantly  reduce 
the  users’  understanding  of  the  model  since  they  do  not  directly  use  or  reference  the 
atomic  data  elements. 


3.1.3  "AS-IS"  PM  and  Simulation 


The  "AS-IS"  PM  was  developed  as  part  of  the  ITI-ALC  program  for  two  reasons.  First,  the 
process  model  provided  a  tool  to  analyze  the  performance  of  the  current  depot  maintenance 
process.  This  dynamic  view  into  the  current  process  helped  to  identify  and  quantify  processing 
bottlenecks  and  to  help  define  the  performance  requirements  that  must  be  overlaid  onto  the 
functional  requirements  to  fully  specify  the  ITI-ALC  system.  By  providing  these  added 
capabilities,  the  "AS-IS"  PM  enhanced  the  ITI-ALC  Business  Process  Reengineering  (BPR), 
supplemented  the  validation  of  the  ITI-ALC  Architecture  baseline,  and  enhanced  ITI-ALC 
system  requirements  definition.  Second,  the  "AS-IS"  PM  provided  a  basis  for  evaluating  the 
IDEF3  concept,  the  capabilities  of  ProSim,  the  translation  from  ProSim  to  a  simulation  model 
using  WITNESS,  and  the  value-added  by  using  process  and  simulation  modeling  within  a  BPR 
effort. 

To  support  the  intended  goals  of  the  program  requirements,  the  IDEF3  technique  was  applied 
according  to  the  specifications  defined  in  InforTtiation  Integration  for  Concurrent  Engineering 
(IICE)  IDEF3  Processing  Description  Capture  Method  Report  (AL-TR- 1992-0057),  dated  May 
1992. 

Because  the  requirement  to  include  the  PM  as  part  of  the  ITI-ALC  program  was  initiated  after 
most  of  the  FM  development  was  completed,  the  development  of  the  "AS-IS"  and  "TO-BE"  PMs 
were  developed  concurrently,  with  the  "AS-IS"  started  slightly  before  the  "TO-BE  .  Therefore, 
while  the  discussion  of  these  two  models  is  presented  in  different  sections  of  the  document,  the 
development,  validation,  and  analysis  were  accomplished  jointly. 

3.13.1  "AS-IS"  PM  and  Simulation  Development 

The  IDEF3  model  was  developed  for  the  purpose  of  representing  process  flow  among  die 
activities  performed  within  depot  maintenance.  Since  the  "AS-IS"  FM  identifies  the  activities 
performed  in  the  depot  maintenance  process  and  the  informational  relationships  among  the 
activities,  the  "AS-IS"  FM  was  used  as  the  starting  point  for  developing  the  "AS-IS"  PM. 


The  "AS-IS"  PM  was  developed  as  a  hierarchical  decomposition  using  a  one-for-one  mapping 
between  the  FM  and  PM  activities  to  produce  the  PM  activity  set.  Using  both  the  "ASJS"  FM 
and  the  "AS-IS"  PM,  each  activity  in  the  activity  set  was  assigned  a  performance  time  and 
branching  conditions  for  each  product  or  output  produced  by  the  activity.  This  process  flow 
information  was  then  overlaid  onto  the  set  of  activities  to  form  the  process  flow. 

Once  the  "AS-IS"  PM  was  developed  and  the  performance  information  documented,  the  "AS-IS" 
PM  was  translated  into  a  simulation  model  to  form  the  basis  for  analyzing  the  operational 
performance  of  the  current  depot  maintenance  process.  This  transition  from  the  IDEF3  model 
into  the  simulation  model  was  intended  to  be  performed  in  accordance  with  the  instructions 
provided  by  the  ProSim  and  WITNESS  tools.  However,  due  to  interface  problems  that  were 
encountered,  a  significant  amount  of  data  manipulation  had  to  be  performed  to  accomplish  a 
basic  translation. 
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3.1.3.2  "AS-IS”  PM  and  Simulation  Validation 

The  validation  of  tiie  "AS-IS"  PM  was  implemented  as  a  combination  of  data  collection,  "AS-IS" 
PM  validation,  and  simulation  model  validation.  The  validation  of  "AS-IS"  PM  used  a  set  of 
workshops  to  walk  the  functional  experts  through  the  PM  network  and  simulation,  requesting 
them  to  validate  and  add  performance  information  as  they  were  able.  These  functional  areas 
experts  consisted  first  of  those  within  the  m-ALC  team  then  the  ARINC  personnel  from  the 
ALCs. 

In  addition  to  providing  model  validation,  the  walkthroughs  also  trained  the  ARINC  personnel  to 
perfonn  the  validation  and  data  collection  process  with  depot  maintenance  personnel  at  their 
respective  ALCs.  The  information  received  was  used  to  further  refined  the  "AS-IS"  PM  and 
simulation  model. 

Because  one  of  the  primary  goals  of  developing  the  "AS-IS"  PM  and  was  to  evaluate  the 
effectiveness  of  IDEF3,  as  implemented  by  ProSim,  the  initial  presentation  format  used  for  this 
model  was  that  provided  by  ProSim.  This  representation  was  supplemented  with  the 
performance  data  presented  in  a  table  format. 

To  complete  the  validation  of  the  "AS-IS"  PM  and  simulation  model,  visits  were  made  to  the 
Oklahoma  City,  Wamer-Robins,  and  Sacramento  ALCs.  During  the  Oklahoma  City  ttip,  the 
same  validation  approach  was  used  as  with  the  ARINC’s  functional  area  experts.  Specifically, 
both  the  "AS-IS"  PM  and  the  "AS-IS"  FM  were  referenced  in  discussing  the  process  flow. 
Unlike  the  initial  validation,  however,  the  results  were  marginal  due  to  the  confusion  caused  by 
addressing  multifonns  of  information  at  one  time. 

For  the  Wamer-Robins  and  Sacramento  trips,  the  presentation  approach  was  modified  such  that 
only  the  performance  information  was  discussed  in  the  context  provided  by  a  brief  explanation  of 
the  activities.  Using  this  approach,  the  validation  was  completed  effectively  and  the  simulation 
again  was  accepted  as  being  realistic. 

Following  these  trips,  the  "AS-IS"  PM  and  simulation  models  were  refined  as  required. 

The  "AS-IS"  PM  was  documented  using  a  page-pair  format  with  a  diagram  being  on  the  lower 
page  and  the  performance  table  for  the  activities  on  the  diagram  presented  on  the  upper  page. 

The  presentation  of  the  simulation  was  accomplished  via  graphic  capability  provided  through 
WITNESS.  For  all  persormel,  this  approach  was  sufficient  for  what  they  needed  to  feel 
comfortable  with  the  operational  validity  of  the  simulation. 

3.1.3.3  ’’AS-IS”  PM  and  Simulation  Traceability 

Since  the  "AS-IS"  PM  was  developed  based  on  the  lowest-level  activities  in  the  AS-IS  FM,  the 
traceability  was  basically  a  one-for-one  correlation  between  the  activities  on  the  two  models. 
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3.1.3.4  “AS-IS”  PM  and  Simulations  Possible  Recommendations  and  Lessons  Learned 

The  requirement  to  use  IDEF3  and  simulation  analysis  within  the  ITI-ALC  program  was  (1)  to 
provide  a  test  case  for  using  the  IDEF3  and  simulation  techniques,  and  (2)  to  evaluate  and  use 
these  performance  analysis  capabilities  to  produce  improved  requirements  and  specifications  for 
the  streamlined  depot  maintenance  process  and  the  ITI-ALC  system.  With  respect  to  these 
objectives,  the  following  possible  recommendations  and/or  lessons  were  learned. 

•  Simulation  is  worth  the  effort. 

The  concept  of  applying  IDEF3  and  WITNESS  to  a  BPR  effort  provides  significant  benefits. 
Using  the  tools  as  stated  by  the  IDEF3  specifications  and  obtaining  these  benefits  m  total  was 
a  challenge.  However,  based  on  this  application  experience  and  the  specifications  for  IDEF3 
and  ProSim  specifically,  the  following  is  a  list  of  possible  recommendations  and/or  lessons 
that  were  learned  from  developing  the  "AS-IS"  PM  and  the  corresponding  simulation. 

.  ProSim  did  not  implement  the  mEFs  rules  completely  as  they  were  specified  in 
Information  Integration  for  Concurrent  Engineering  (IICE)  IDEF3  Process  Descnption 
Capture  Method,  dated  May  1992. 

The  IDEF3  specification  identifies  five  object  states  available  throughout  the  project.  The 
Entity  Description  Type  is  a  pooled  item  in  ProSim  and  can  only  be  set  to  one  type  for  the 
entire  project.  The  IDEF3  specification  identifies  junctions  as  providing  a  mechanism  to 
specify  the  logic  of  process  branching.  ProSim  junctions  expand  beyond  branching  logic  md 
contain  much  of  the  functionality  required  by  processes,  such  as  creating  objects  or  passing 
multiple  created  objects. 

•  The  translation  from  the  process  description  to  the  simulation  was  one-directional. 


The  intent  of  IDEF3,  and  implemented  via  ProSim,  is  to  collect  process  flow  and  perform^ce 
information  so  as  to  facilitate  the  development  and  execution  of  a  simulation  model  within 
WITNESS.  Following  the  translation  from  ProSim  to  WITNESS,  a  significant  amount  of 
effort  is  required  to  adjust  and  enhance  the  information  in  WITNESS  before  the  simulation 
model  is  operational  and  available  for  verification,  validation,  and  analysis. 

Because  of  this  additional  work  required  within  WITNESS,  and  because  the  adjustments 
made  in  WITNESS  can  not  be  transferred  back  to  ProSim,  the  benefits  of  the  IDEF3/ProSim 
representation  and  capabilities  are  lost  once  the  first  translation  occurred. 

During  the  ITI-ALC  program,  this  situation  was  addressed  by  maintaining  two  separate  but 
correlated  models.  The  IDEFs/ProSim  representation  was  maintained  for  display  of  the 
network  while  the  WITNESS  model  was  maintained  for  exercising  the  simulation. 
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•  The  process  model  notation,  as  represented  by  ProSim,  was  not  an  effective 
presentation  vehicle. 

The  information  represented  on  a  process  flow  network  is  very  limited,  reducing  its 
effectiveness  as  a  communication  and  analysis  tool.  To  read  and  rmderstand  a  network 
requires  a  labor  intensive  process  of  obtaining  and  integrating  information  from  other 
sources. 

The  readability  of  an  IDEF3  network  could  be  significantly  increased  if  the  process  flow 
information  could  be  overlaid  onto  the  network.  For  example,  labels  should  be  placed  on  the 
relationship  arrows,  conditions  listed  with  the  branches,  and  timing  and  resource 
requirements  placed  a  process. 

To  support  the  presentation  and  readability  of  the  IDEF3  process  flows  for  the  ITI-ALC 
program,  the  functional  models  and  performance  data  worksheets  were  used  to  ^scuss  and 
validate  the  process  flows.  This  approach  improved  the  communications,  but  using  just  the 
performance  worksheets  proved  to  be  the  most  effective  for  discussing  and  validating  the 
process  flows. 

•  Scenarios,  or  small  snippets  of  the  entire  process,  help  support  the  presentation  of  the 
process,  but  did  not  provide  the  foundation  necessary  to  effectively  analyze  the 
performance  of  the  depot  maintenance  process. 

The  IDEF3  concept,  and  specifically  the  ProSim  tool,  was  developed  based  on  the  idea  that  a 
large  process  flow  could  be  developed  and  analyzed  as  small  scenarios  or  snippets  of  the 
entire  process.  These  small  scenarios,  consisting  of  five  to  ten  processes,  could  then  be 
presented  on  a  single  page. 

This  segmented  approach,  however,  limits  the  capability  to  evaluate  the  impacts  caused  by 
changes  across  the  entire  process  flow.  Evaluating  the  entire  process  as  a  unit  was  especially 
important  when  trying  to  identify  operational  bottle  necks  when  parameters  in  one  segment 
was  changed. 

•  The  data  collection  for  the  "AS-IS”  PM  and  the  simulation  would  have  been 
approached  differently  if  this  modeling  and  analysis  requirement  had  been  established 
at  the  beginning  of  the  ITI-ALC  program. 

The  performance  information  required  to  develop  the  "AS-IS"  PM  would  have  been  collected 
as  part  of  the  data  collection  for  the  "AS-IS"  FM. 

3.2  ”TO-BE”  MODELS 

Through  the  analysis  of  the  "AS-IS"  models,  process  improvement  concepts  were  developed.  To 
provide  the  foundation  for  representing  and  evaluating  these  improvement  concepts,  a  set  of 
"TO-BE"  models  was  developed  and  refined.  These  "TO-BE"  models  include  the  functional, 
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data,  and  process/simulation.  In  addition,  the  System  Model  (SM)  which  represente  a  concept  for 
the  ITI-ALC  system,  is  also  a  “TO-BE”  model  and  will  be  discussed  m  Section  4.4  of  this 

document. 

3.2.1  "TO-BE”  Functional  Model 

The  "TO-BE"  FM  represents  an  improved,  future  view  of  the  Air  Force’s  aircraft  PDM  process 
based  on  the  integration  of  all  the  process  improvement  ideas  identified  thus  fc.  Once  t^mple^ 
the  "AS-IS”  FM  provided  the  foundation  for  transiating  the  various  aspects  ot  the 

concept  into  the  "TO-BE”  DM  and  "TO-BE"  PM,  ^d  also 

of  ITI-ALC  system  requirements  used  in  the  development  of  the  SSS.  The  T 

provided  the  foundation  for  developing  the  functional  requirements  needed  to  identify  and 

Laluate  the  potential  technologies  for  implementing  the  "TO-BE"  depot  maintenance  process. 

3.2.1.1  "TO-BE"  FM  Development 

The  first  step  was  to  review  the  “AS-IS”  FM  purpose  and  viewpoint  to  ensure  the  user  remained 

the  focal  point  for  the  program.  The  “TO-BE”  FM  viewpoint  was  developed  so  as  not  to  restact 

any  implementation  cLepts  and  be  flexible  enough  to  allow  for  the 

needed  for  the  future  view  of  depot  maintenance.  Dunng  the  development  of  th 

model  all  restrictions  were  removed  that  were  related  to  implementation  limitations  based  on 

existing  policies  and  rules  and  technology  limitations,  while  the  incorporation  of  innovative  ideas 

and  removing  the  “this  is  the  way  it  has  always  been  done”  attitude  were  encouraged. 

The  “TO-BE”  FM  development  was  performed  as  a  top-down  and  a  bottom-up  an^ys^  of  the 
^S-IS’’ model,  based  to  a  great  extent  on  the  arrow  patterns  of  the  “AS-IS”  modd.  The  top- 
do™  approach  rvas  used  to  understand  the  context  of  the  entire  system  as  weU  as  ttot  for  ^h 
activity  The  bottom-up  analysis  aliowed  for  the  identification  of  specific,  focused  problems 
identified  through  the  analysis  of  each  activity,  and  which,  through  use  of  impact  analysis  on 
Other  functions,  the  larger  problem  and  possible  cause  of  the  problem. 

To  these  improvement  opportunities  were  added  the  problems  and  mprovement  suggestions 
provided  by  the  ITI-ALC  team's  functional  experts  and  users  at  the  ALCs,  the  data  improvemen 
opportunities  identified  from  the  "AS-IS"  DM,  and  the  process  ^rformance  problem  are^ 
identified  via  the  "AS-IS"  PM.  The  information  provided  by  functional  experts  and  users  was 
often  based  on  situations  and  corrections  that  individuals  implemented  to  cucumyent  Problems 
they  encountered  with  the  depot  maintenance  process.  While  ^s  information  proyi 
sigdficant  insight  into  the  process,  the  ideas  had  to  be  analyzed  to  dete^e  how  they  shou  d 
beTbe  represented  in  the  "TO-BE"  FM.  Specifically,  the  problems  had  to  be  ^n^y^ed  to 
determine  if  they  were  indeed  problems  or  if  they  were  symptoms  of  otiier  problems.  Similarly 
improvement  ideas  had  to  be  analyzed  to  ensure  they  were  correcting  a  real  problem  or 
symptom,  and  if  the  inclusion  of  the  improvement  would  have  a  det^entd  ^ 

^cts  of  the  process.  As  interfacing  information  systems  were  identified,  they  were  identifie 

as  mechanism  for  those  functions  requiring  the  interfacing. 
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In  conjunction  with  the  development  of  the  "TO-BE"  FM,  the  BPIs  being  incorporated  into  the 
"TO-BE"  FM  were  specifically  identified  and  listed,  with  a  short  narrative  generated  for  each. 

3.2.1.2  "TO-BE"  FM  Validation 

Because  the  "TO-BE"  FM  represented  a  vision  for  an  improved,  streamlined  depot  maintenance 
process  that  will  be  implemented  in  the  future,  there  was  no  conclusive  way  to  validate  the 
process  represented  by  the  "TO-BE"  FM.  However,  the  "TO-BE"  FM  was  validated  to  the  extent 
that  all  members  of  the  ITI-ALC  team,  and  especially  the  functional  experts  and  users,  accepted 
the  "TO-BE"  concept  because  they  believed  the  model  to  represent  a  practical  and  beneficial 
concept  for  PDM.  Therefore,  the  validation  of  the  "TO-BE"  FM  was  implemented  to  provide  the 
functional  experts  and  users  wifii  a  complete  understanding  of  the  "TO-BE"  process  concept  and 
to  gain  their  acceptance  of  the  improved  process. 

Based  on  these  objectives,  the  validation  of  the  "TO-BE"  FM  was  performed  by  developing  a  set 
of  scenarios  through  the  model  that  specifically  addressed  the  BPIs.  Using  these  scenarios, 
walkthroughs  of  the  model  were  presented  to  members  of  the  ITI-ALC  team,  followed  by 
discussion  of  the  BPIs.  Following  the  internal  validations,  the  refined  information  was  presented 
to  ARINC  personnel  from  the  ALCs,  but  with  increased  emphasis  on  the  BPIs  rather  that  the 
"TO-BE"  FM.  Finally,  the  BPIs  were  presented  to  the  users  firom  the  ALCs.  Following  each 
validation  step,  the  comments  received  were  used  to  refine  the  "TO-BE"  FM. 

3.2.1.3  "TO-BE"  FM  Traceability 

The  primary  traceability  for  the  "TO-BE"  FM  was  back  to  the  "AS-IS"  FM.  This  traceability 
ensured  that  each  activity  in  the  "AS-IS"  FM  was  accounted  for  in  the  "TO-BE"  FM,  either  in 
terms  of  being  included,  identified  as  not  being  included  because  it  was  a  non-value-added 
activity  and  therefore  not  included  in  the  "TO-BE"  FM,  or  identified  as  a  new  activity  within  the 
"TO-BE"  FM. 

3.2.1.4  “TO-BE  FM  Possible  Recommendations  and  Lessons  Learned 

The  following  possible  recommendations  and/or  lessons  were  learned  during  the  development 
and  use  of  the  "TO-BE"  FM. 

•  Effective  scoping  of  the  functional  model  is  important  to  maximize  the  benefits  received 
from  a  BPR  effort. 

The  focal  point  of  the  ITI-ALC  program  was  the  PDM  mechanics  vd^n  the  Air  Force's  Aw 
Logistics  Centers.  However,  the  scope  was  expanded  at  program  iiutiation  to  include  within 
the  scope  of  ITI-ALC's  BPR  analysis  the  operational  environments  impacting  the  mechanic's 
performance  effectiveness.  This  expanded  scope  provided  the  foundation  needed  to 
effectively  streamline  the  mechanic  process  since  much  of  the  work  performed  by  the 
mechanic  was  a  duplication  or  continuation  of  the  work  performed  by  other  individuals. 
Therefore,  restricting  the  scope  of  the  ITI-ALC  program  just  to  the  mechanic  would  have 
limit  the  BPIs  identified  for  the  mechanic  and  would  have  reduced  improved  effectiveness 
provided  by  the  proposed  ITI-ALC  system. 
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•  By  including  the  cursory  look  at  the  engine  and  component  environments  allowed  for 
the  understanding  for  how  the  ITI-ALC  system  could  aid  the  entire  depot  maintenance 
process. 

While  the  focus  of  the  ITI-ALC  program  was  the  PDM  mechanic,  the  basic  activities 
performed  by  all  personnel  throughout  the  depot  have  much  in  common  and  are  tightly 
integrated.  Making  the  minimal  effort  required  to  imderstand  a  larger  perspective  of  the 
depot  operations  in  terms  of  the  commonality  and  integration  provided  the  ITI-ALC  team 
with  the  necessary  background  to  maximize  the  benefits  received  from  the  improved  depot 
maintenance  process  and  the  implementation  of  the  ITI-ALC  system.  Therefore,  using  this 
broader  perspective  assured  that  the  improvement  concepts  addressed  all  common  aspects 
and  assured  that  recommended  improvements  did  not  negatively  impact  other  processes  and 
personnel  involved  in  depot  maintenance. 

3.2.2  ”TO-BE"  Data  Model 

The  “TO-BE”  DM  defined  the  maintenance  information  requirements  needed  to  support  the  ITI- 
ALC  system  and  established  the  foundation  for  implementing  the  physical  database. 

3.2.2.1  “TO-BE”  DM  Development 

The  "TO-BE"  DM  was  developing  using  the  "AS-IS"  DM  and  the  information  specified  on  the 
interfaces  of  the  "TO-BE"  FM.  The  "AS-IS"  structure  was  then  adjusted  and  refined  to  reflect 
the  information  represented  on  the  "TO-BE"  FM  and  the  BPI  concepts  that  had  been  identified. 
One  of  the  key  differences  in  the  approach  taken  by  the  “TO-BE”  DM  versus  the  “AS-IS”  DM 
was  to  represent  an  information  system  that  may  hold  data  pertaining  to  more  than  one  end-item 
and  to  more  than  one  ALC  without  duplication  of  data.  Current  systems  tend  to  segregate  data 
by  weapon  system  and  ALC.  The  creation  of  data  structures  that  were  sufficiently  complex  to 
represent  these  interrelationships  among  the  data  and  the  addition  of  the  technical  information 
represented  by  the  lETMs  expanded  the  size  of  the  “TO-BE”  DM  as  compared  to  the  “AS-IS” 
DM. 

The  availability  of  specific  information  about  fiiture  data  systems  and  the  details  of  data 
interfaces  for  systems  currently  imder  development,  such  as  DMMIS,  was  limited.  Therefore,  the 
details  that  were  delineated  for  the  data  interfaces  between  the  ITI-ALC  system  and  these  types 
of  systems  were  limited.  As  these  systems  are  developed  and  documentation  generated,  the 
interface  and  internal  data  structure  definition  can  be  evaluated  relative  to  the  ITI-ALC  system 
implementation. 

3.2.2.2  “TO-BE”  DM  Validation 

The  "TO-BE"  DM  was  validated  in  two  ways.  The  first  validation  step  was  based  on  its  close 
relationship  with  the  validated  "TO-BE"  FM.  The  information  for  the  "TO-BE"  DM  was 
obtained  from  the  validated  "TO-BE"  FM.  Therefore,  by  tracing  the  information  identified  in  the 
data  model  to  the  functional  model,  the  content  of  the  data  model  was  validated.  The  second 
validation  step  was  accomplished,  by  walking  through  the  subject  areas  with  members  of  the  ITI- 
ALC  team  and  subject  matter  experts.  Because  the  “TO-BE”  DM  represented  a  projected  rather 
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than  an  existing  maintenance  environment,  the  second  validation  step  was  based  on  a  consensus 
by  subject  experts  that  all  data  needed  to  perform  depot  maintenance,  including  that  needed  to 
implement  the  BPIs,  were  specified. 

The  "TO-BE"  DM  was  documented  within  the  Architecture  Report.  To  facilitate  the 
documentation,  the  model  was  separated  into  subject  areas,  each  of  which  addressed  a  specific 
aspect  of  the  data  within  the  model.  These  subject  areas  were  Materiel,  Plan,  Technical 
Information,  Action,  Facility,  Location,  Organization,  and  Person.  Larger  subject  areas  were 
further  subdivided  into  sub-subject  areas.  An  entity  versus  subject  area  matrix  was  developed  to 
aid  in  identifying  the  subject  areas  in  which  each  entity  was  located. 

Within  the  Architecture  Report,  the  "TO-BE"  DM  was  presented  at  two  levels  of  detail.  To 
provide  a  overview  of  the  "TO-BE"  DM,  the  entire  Entity  Relationship  Diagram  (ERD)  was 
presented  as  a  unit.  To  provide  more  detail  and  usability,  the  ERDs  for  each  subject  areas  were 
presented  along  with  their  narratives  using  a  page-pair  format.  A  glossary  of  definitions  for  the 
entities  and  attributes  was  developed  and  documented  in  combination  with  the  "TO-BE"  FM 
model  glossary. 

3.2.23  “TO-BE”  DM  Traceability 

The  traceability  of  the  "TO-BE"  DM  was  developed  back  to  "TO-BE"  FM.  This  traceability 
consisted  of  mapping  the  entities  in  the  "TO-BE"  DM  to  the  arrows  associated  with  the  lowest- 
level  nodes  containing  the  information  within  the  "TO-BE"  FM.  This  tracing  was  documented  in 
the  RTC. 


3.2.2.4  “TO-BE”  DM  Possible  Recommendations  and  Lessons  Learned 

The  following  possible  recommendations  and/or  lessons  were  learned  during  the  development 

and  use  of  the  “TO-BE”  DM. 

•  The  ERWin  tool  had  significant  limitations  with  respect  to  producing  quality  looking 
documentation. 

Models  are  developed  for  use  as  analysis  tools,  therefore,  they  must  be  documented  and 
presented  in  a  maimer  that  facilitates  the  analysis  process.  While  the  ERWin  tool  facilitated 
the  development  by  controlling  the  migration  of  attributes  among  the  entities,  the  tool  did  not 
allow  for  user  control  of  the  relationship  layout  among  the  entities.  This  resulted  in  data 
model  that  was  difficult  and  time  consuming  to  read  and  analyze.  To  correct  this  situation, 
the  completed  ERWin  model  was  exported  to  the  Design/IDEF  tool  which  allowed  the 
developer  to  rearrange  the  entities  and  relationships  so  as  to  maximize  the  readability  and 
usefulness  of  the  model. 
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3.2.3  "TO-BE"  PM  and  Simulation 

The  "TO-BE"  PM  and  simulation  were  developed  to  analyze  the  operational  or  dynamn^pecte 
the  "TO-BE"  depot  maintenance  process  as  represented  in  the  "TO-BE"  FM  concept.  Through 
the  operational  perspective,  the  "TO-BE"  process  flow  concepts  were  evaluated,  the  processing 
times  were  optimized  to  maximize  the  efficiency  of  the  process,  the 

for  the  resources  were  established,  the  predicted  improvement  potential  from  the  ^S-IS  to  fte 
"TO-BE"  environment  was  estimated,  and  the  support  structure  was  establishe  or  pe  g 

the  performance-technology  cost  trade-off  analysis. 

3.2.3.1  "TO-BE"  PM  and  Simulation  Development 

The  development  of  the  "TO-BE"  PM  was  based  primarily  on  the  structure  of  the  "TO-BE"  FM^ 
This  translLn  of  information  was  obtained  by  establishing  a  one-for-one  correl^ion  of  the 
activities  from  the  functional  model  to  the  process  model.  The  process  flow  among  Ae  activi 
was  established  by  analyzing  the  interface  relationships  between  the  functional 
Z  Sing  the  sar^e  sceLos  developed  for  the  "AS-IS"  PM  These  scenarios  ^eluded  ^ 
small  scenarios  directed  at  the  mechanic  and  the  planner  and  a  larger  scenario  directed  at  toe 
mechanics  performing  their  complete  PDM  process.  By  developmg  corresponding  scenanos 
between  toe  "AS-IS"  and  "TO-BE"  process  definitions,  toe  baseline  was  established  for 
predicting  toe  benefits  received  from  implementing  toe  improved  depot  maintenance  process. 

Throughout  toe  development  of  toe  process  flows,  a  close  working  relationship  w^  establish^ 
among  toe  PM/simulation  developers,  toe  ITI-^C  team’s  frmctional 

developer,  and  toe  "TO-BE"  FM  developer.  When  reasonably  stable,  the  TO-BE  and  toe 

"TO-BE"  FM  were  used  as  guidelines  for  developing  toe  performance  timing  or  ea 
activities  in  the  process  model  network.  Hus  performance  time  was  predicted  on 

corresponding  times  within  the  ”AS-1S”  PM,  on  the  expertise  of  functional  experts,  and  on  the 
performance  capabilities  of  technologies. 

When  toe  "TO-BE"  PM  was  completed,  it  was  translated  into  toe  "TO-BE"  simulation  model. 
While  toe  "AS-IS"  PM  described  the  flow  of  information  and  matenal  ^^ngh  the  epo 
maintenance  process,  it  provided  only  limited  insight  in  toe  actual  opemtion  of  toe  TO-Bh 
depot  mainteLice  process.  The  "TO-BE"  simulation  model  provided  toe  capability  to  pretoc 
toe  operational  cha^cteristics  of  toe  "TO-BE"  depot  maintenance  process. 
was  developed  by  combining  toe  performance  characteristics  with  toe  TO-BE  PM.  the 
simulation  Ldel  was  then  exercised  to  evaluate  toe  effectiveness  of  toe  proposed  process 

improvements. 

3.2.3.2  "TO-BE"  PM  and  Simulation  Validation 

Due  to  toe  late  time  frame  in  which  toe  PM  effort  was  performed,  the  validation  of  toe  "AS-IS" 

and  "TO-BE"  PMs  were  performed  concurrently.  The  process 

The  materials  used  to  accomplish  toe  validation  were  toe  TO-BE  PM, 
simulation  model,  toe  performance  information  for  each  of  toe  activities,  and  toe  - 
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The  first  level  of  validation  was  performed  by  the  ARINC  functional  experts  firom  each  of  the 
ALCs.  During  these  reviews,  the  process  models  were  used  by  subject  matter  experts  ^  the  basis 
to  discuss  the  "TO-BE"  process  flow  and  die  associated  performance  formation.  The 
simulation  models  were  also  demonstrated  to  show  that  the  individual  task  times  integrated  to 
produce  the  major  flow  days  they  were  a  custom  to  seeing. 

The  second  level  of  validation  was  performed  using  the  same  approach  at  the  Oklahoma  City 
ALC.  The  benefits  received  firom  this  validation  were  viewed  as  minimal.  To  adjust  for  the  next 
validations  at  Wamer-Robins  and  Sacramento  ALCs,  the  validation  process  was  modified  so  as 
to  discuss  only  the  performance  information  and  to  demonstrate  the  simulation  model,  without 
presenting  or  discussing  the  "TO-BE"  PM  network.  This  approach  was  viewed  as  being  effective 
and  successful. 

For  documentation  within  the  Architecture  Report,  the  complete  process  flow  model  was  too 
large  to  present  effectively  as  a  single  item.  Therefore,  the  model  was  segmented  to  reflect  the 
decomposition  of  activities  represented  in  the  "TO-BE"  FM.  Each  diagram  w^  m 

page-pair  format  with  the  corresponding  page  containing  the  link  back  to  the  "TO-BE  FM,  the 
activity  name,  a  short  activity  description,  the  resources  performing  activity,  and  the  products 
produced  by  each  activity. 

3.2.33  "TO-BE"  PM  and  Simulation  Traceability 

Documented  in  the  "TO-BE"  PM  page-pair  format,  the  "TO-BE"  PM  was  traceable  back  to  the 
"TO-BE"  FM,  lining  basically  a  one-for-one  correlation  between  the  activities  in  the  two  model. 
This  tight  correlation  exists  because  of  the  manner  in  which  the  "TO-BE"  PM  was  developed 
from  the  "TO-BE"  FM. 

3.2.3.4  "TO-BE"  PM  and  Simulation  Possible  Recommendations  and  Lessons  Learned 
Becaiise  the  developments  of  the  "AS-IS"  and  "TO-BE"  PMs  and  simulations  were  Perfome^  to 
a  great  extent  in  parallel,  the  possible  recommendations  and  lessons  learned  for  the  TO-BE 
development  are  the  same  as  those  listed  in  the  "AS-IS"  PM  section  of  this  report. 

3.3  SYSTEM  MODEL  (SM) 

The  SM  represented  an  operational  concept  for  the  ITI-ALC  system  and  provided  the  transition 
from  the  "TO-BE"  functional,  data,  process,  and  simulation  models  to  the  high-level  design 
documented  in  the  SSDD.  In  defining  tiiis  operational  concept,  the  SM  identified  the  system 
hierarchy.  Computer  Software  Configuration  Item-to-Computer  Software  Configuration  Item 
(CSCI-to-CSCI)  interfaces,  internal  CSCI  interfaces,  external  interfaces,  and  inputs  and  outputs 
of  each  process.  As  represented  in  Figure  3-1,  the  SM  was  used  to  directly  develop  system 
design  sections  which  are  the  major  and  most  important  components  of  the  SSDD. 
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Figure  3-1.  Overall  Relationship  of  Three  Architecture  Models 
to  the  Design  of  the  ITI-ALC  System 


3.3.1  SM  Development 

A  data  flow  diRgr^rnming  methodology  was  selected  to  represent  the  SM  because  of  the  need  for 
a  graphic  language  to  represent  information  (data)  as  it  circulates  (flows)  and  is  transfoimed 
throughout  a  system.  The  graphics  flow  provided  the  users  and  developers  with  an  effective 
means  of  understanding  and  evaluating  complex  system  requirements  in  a  simple  and  accurate 
language.  The  graphics  flow  representation  did  not  include  any  implementation  concepts, 
thereby  not  forcing  any  implementation  decisions. 

As  represented  in  Figure  3-2,  analysis  of  the  various  operational  scenarios  and  the  grouping  of 
functions  and  capabilities  required  to  operate  within  those  scenarios  were  used  to  define  modes 
or  high-level  structure  of  the  ITI-ALC  system  as  represented  in  the  SM.  Having  formed  this  high 
level  structure,  fire  "TO-BE"  functional  and  data  models  were  analyzed  to  develop  an 
understanding  of  the  operational  requirements  of  the  I  ITALC  system. 

The  "TO-BE"  FM  defined  the  activities  to  be  performed  within  the  ITI-ALC  system,  and  the 
functional  interface  requirements  with  both  the  users  of  the  ITI-ALC  system  and  the  information 
systems  external  to  the  ITI-ALC  system.  The  "TO-BE"  DM  defined  the  relationships  or  busmess 
rules  among  the  data  elements  and  the  data  stores  that  will  be  maintained  within  the  ITI-ALC 
system  and  will  require  an  interfacing  capability.  Data  stores  are  defined  as  an  aggregate  of  data 
the  system  must  “remember”  for  a  period  of  time.  When  the  system  is  fully  implemented,  these 
data  stores  will  typically  exist  as  files  or  databases. 
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Figure  3-2.  FM/DM-to-SM  Translation 

The  SM  was  developed  using  a  top-down  hierarchical  approach  that  corresponded  to  the 
structure  of  the  "TO-BE"  FM.  The  hipest  level  diagram  contained  one  faction  that  represented 
the  SM  along  with  a  general  description  of  the  interfaces  with  external  information  system  and 
with  the  user.  On  subsequent  diagrams,  each  function  was  detailed  out  for  these  interfaces  along 
with  the  inclusion  of  the  interfaces  to  the  internal  data  stores  identified  through  the  "TO-BE" 

DM. 

Included  in  the  transition  from  the  "TO-BE"  FM  and  "TO-BE"  DM  was  the  elimination  of 
organizational  and  functional  redundancies,  and  the  addition  of  system-level  functions  and  data 
to  support  the  operational  functions.  Organizational  redundancies  result  when  the  same 
function  is  perfomed  at  two  different  levels  of  maintenance.  Although,  from  an  end-user 
perspective  these  two  processes  are  done  at  different  times,  a  system  that  would  support  these 
two  maintenance  tasks  would  need  only  one  software  process  to  handle  both.  Functional 
redundancies  are  very  similar  to  organizational  redimdancies  and  a  common  process  that  could 
support  all  of  the  common  functionality,  regardless  of  where  it  is  performed  within  the  depot 
maintenance  process.  Adding  system-level  functions  and  data  to  support  operational  functions 
is  done  by  determining  what  ITI-ALC  computer  system  functions  will  be  performed  to  support 
the  reduced  set  of  FM  functions  that  resulted  from  the  above  elimination  of  redundancy.  Items 
such  as  display,  input,  and  ou^ut  from  the  SM  are  added  to  the  list  of  functions  to  be  performed 
as  part  of  the  system  capabilities. 
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3.3.2  SM  Validation 


The  SM  validation  process  ensured  the  SM  provided  a  complete  and  accurate  representation  of 
the  m-ALC  system.  Because  the  “TO-BE”  FM  controls  the  baseline  processes  that  m^t  be 
supported  by  the  system  represented  by  the  SM,  the  validation  of  the  FM  provided  a  si^ficant 
step  toward  the  validation  of  the  SM.  Beyond  this  indirect  validation,  the  complete  validation  of 
the  SM  was  performed  as  a  two  step  process  consisting  of  a  review  process  involving  the  ITI- 
ALC  team  which  included  functional  experts  and  a  validation  process  mvolving  the  i^ers. 
Following  each  step  in  the  validation,  the  model  was  updated  based  on  an  analysis  of  the 
comments  received.  The  SM  review  process  was  addressed  from  two  Perspectives  *e 

review  ensured  (1)  correlation  between  the  SM  and  the  “TO-BE”  versions  of  the  FM  and  DM, 
(2)  confirmation  between  the  SM  and  the  syntax  of  the  modeling  language,  (3)  completion  of  the 

glossary  and  (4)  maximized  correlation  with  other  depot  maintenance  information.  Second,  toe 

review  ensured  toe  model  provided  an  understandable  representation  such  that  an  effective 
validation  process  could  be  performed. 


The  review  process  included  two  walkthroughs  and  one  peer  review,  both  mvolving  internal 
functional  experts  and  systems/software  engineers.  The  first  walkthrough  onented  the  team  to 
the  modeling  technique,  provided  an  overview  concept  of  the  model  structure^d  toe  operation^ 
concept  represented  via  the  model,  and  ensured  that  the  SM  addressed  toe  functional  and  data 
requirements  represented  in  the  "TO-BE”  functional  and  data  models 


The  walkthrough  was  followed  by  one  peer  review  conducted  as  a  desk  top  review.  The  peer 
review,  which  included  functional  experts,  provided  ample  time  for  each  reviewer  to  Mly  review 
and  analyze  the  model.  This  review  ensured  that  the  SM  was  usable,  as  well  as  comprehensive  ^d 
compliant  with  project  needs.  It  also  provided  early  visibility  mto  the  status  and  quality  of  toe 
product,  with  ample  time  to  take  corrective  action  if  progress  and  quality  are  not  at  acceptable 

levels. 


Finally  the  SM  was  validated  with  users  by  using  a  walkthrough  approach  This  review  ensured 
that  the  operational  concept  represented  by  the  SM  accurately  and  completed  addressed  the 
operational  requirements  of  depot  maintenance  personnel. 


The  SM  was  documented  using  a  pair-pair  format  accompanied  by  a  node  list  and  a  glossary. 
The  node  list  was  an  indentured  list  of  each  activity  contained  in  the  model.  The  diagram  portion 
of  the  model  was  presented  in  a  page-pair  format  with  a  diagram  presented  on  the  loiwr  page  and 
a  nairative  describing  the  information  and  control  flow  represented  on  the  diagram.  The  glossy 
contained  a  name  for  each  interface  Wentified  in  the  SM,  a  definition  for  toe  mterface,  and  a 
reference  to  the  diagram  on  which  the  interface  appeared. 


3.3.3  SM  Traceability 


traceability  of  the  SM  was  to  the  "TO-BE"  FM  to  ensure  that  each  function  in  toe  "TO-BE" 
identified  and  performed  at  least  in  part  by  the  ITI-ALC  system  was  addressed  m  the  SM. 
traceability  was  a  mapping  between  toe  lowest-level  process  in  the  SM  and  toe  lowest-level 
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functions  in  the  "TO-BE"  FM.  The  validation  of  the  linkage  between  these  models  was 
facilitated  by  the  tracing  capability  provided  by  the  RTC. 

3.3.4  SM  Possible  Recommeiidations  and  Lessons  Learned 

During  the  development  of  the  SM  model,  the  following  were  possible  recommendations  or 
lessons  learned  that  were  identified. 

•  Data  flow  diagrams  are  an  exceUent  way  to  represent  the  system  model. 

The  goal  of  the  system  model  is  to  provide  a  transition  from  the  fimctional  and  data 
requirements  identified  via  the  functional  and  data  models  to  the  design  of  the  system  that 
will  satisfy  those  requirements.  Using  data  flow  diagrams  provided  an  effective  way  of 
presenting  this  transition. 

The  data  flow  diagrams  supported  a  top-down  approach  and  provided  a  leveling  of 
information  that  reduced  the  complexity  of  information  on  any  one  diagram  while  allowing 
for  increasingly  more  detailed  information  on  decomposed  diagrams,  "niis  top-down 
approach  corresponded  to  that  used  to  developed  the  functional  model,  resulting  in  a  strong 
correlation  between  the  "TO-BE"  functional  model  and  the  system  model. 

The  flow  diagrams  accommodated  the  specification  of  data  stores  md  ^e  interfaces  ydth 

the  data  stores.  The  definition  of  these  data  stores  provided  an  effective  link  to  the  entities 
specified  within  the  "TO-BE"  data  model. 

The  data  flow  diagrams  accommodated  the  incorporation  of  system  control  activities  and  the 
specification  of  control  flows  among  the  activities  and  the  interfaces  to  the  user,  the  intemd 
data  stores,  and  the  external  database  systems.  Furthermore,  a  side  benefit  of  this  operate  is 
that  superior  cost  estimates  based  on  Function  Points  can  be  derived  from  the  models. 

•  Although  DFDs  are  an  excellent  way  to  represent  the  system  model,  the  full  benefit  of 
the  model  can  not  be  realized  by  those  not  knowledgeable  in  the  model  language. 

Non-native  languages  have  advantages  over  a  native  language  because  they  are  developed  to 
efficiently  present  specific  types  of  information.  Native  languages,  on  the  other  hand  have 
the  advantage  in  that  they  are  common  and  understood  by  a  majority  of  people,  even  though 
they  may  not  present  the  information  as  efficiently  and  accurately.  This  language  difference 
causes  some  problems  because  the  model  developers  prefer  the  non-native  lang^es  while 
system  users  often  prefer  the  native  languages.  To  maximize  the  benefit  of  these  two 
perspectives,  a  two  step  approach  should  be  used  that  enhances  the  commumcations  between 
the  two  situation.  First,  users  must  be  motivated  and  willing  to  take  the  effort  to  gam  a 
reasonable  understanding  of  the  non-native  language  in  order  to  fully  a^ppreciate  the 
information  contained  in  the  model  and  to  fully  utilize  the  model.  Second,  the  dwelopers 
must  be  motivated  and  willing  to  supplement  the  non-native  language  with  an  ICON  level 
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abstraction  to  facilitate  the  understanding  by  the  users.  Development  of  a  common 
imderstanding  and  effective  communication  maximizes  the  effectiveness  and  usefulness  of 
the  information  produced. 


3.4  BUSINESS  CASE  (BC)  CDRL  SEQUENCE  A002 

The  m-ALC  Business  Case,  an  example  of  a  Functional  Economic  Analysis  (FEA),  documented 
predictions  of  the  cost  benefits  that  would  be  realized  from  the  implementation  of  the  improved 
depot  maintenance  process  and  the  ITI-ALC  system  at  the  ALCs. 

The  project  approach  included  requirements  determination,  system  engineering,  and  business 
case  development  as  illustrated  in  Figure  3-3.  The  methodologies  employed  were  consistent  widi 
the  Department  of  Defense’s  Framework  for  Managing  Process  Improvement  as  discussed  in 
DoD  8020. 1-M,  Functional  Process  Improvement.  The  economic  analysis  components  of  the 
approach  followed  the  direction  in  the  DoD  Corporation  Information  Management  (CIM) 
Functional  Economic  Analysis  Guidebook  (January  1993)  and  the  requirements  from  the  Office 
of  Assistant  Secretary  of  Defense’s  (OSD)  Guide  for  Developing  AIS  Cost  and  Operational 
Effectiveness  Analyses  (OSD,  June  1994). 
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Figure  3-3.  ITI-ALC  Approach  to  Business  Case  Development 
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The  bxisiness  case  required  a  number  of  individual  steps  be  performed,  the  results  of  which  were 
integrated  as  part  of  the  business  case  development.  These  steps  included  definition  of  the 
improvement  proposals,  selection  of  the  ALC  site  to  provide  the  basis  for  the  business  case, 
collection  of  the  cost  for  the  technologies  needed  to  implement  the  improvements,  collection  of 
the  cost  information  for  the  selected  site,  and  the  development  of  the  business  case. 


3.4.1  BC  Development 

Like  all  of  the  other  efforts  on  the  ITI-ALC  project,  development  of  the  Business  Case  document 
was  focused  on  the  user,  therefore  the  effort  started  with  data  collection.  To  initiate  the  data 
collection  effort,  the  ITI-ALC  team  reviewed  the  AFMC  and  depot  maintenance  mission, 
objectives,  and  strategy.  Pertinent  Air  Force,  AFMC,  and  DoD  planning  documents  were 
reviewed  and  interviews  conducted  at  each  of  the  five  ALCs  to  identify  function  and  information 
relationships.  During  data  collection  at  the  select  sites,  SM-ALC  and  WR-.^C,  (refer  to  Section 
1.4.2)  manpower  and  cost  information  was  obtained  through  the  financial  management  and 
production  directorates.  The  team  then  associated  resource  consumption  with  each  activity  in  the 
“AS-IS”  Functional  Model  resulting  in  an  activity-based  cost  functional  model  to  the  lowest- 

level. 

To  facilitate  the  implementation  of  the  improvements  identified  through  the  ITI-ALC  program 
and  to  begin  realizing  early-on  benefits  from  implementing  the  improvements,  the  business  case 
was  developed  around  a  set  of  incremental  implementation  concepts.  Based  on  knowledge  of  the 
depot  maintenance  process,  the  ITI-ALC  team  logically  grouped  slices  from  each  BPI  into 
"packages"  called  Process  Improvement  Proposals  (PIPs). 

These  PIPs  offer  a  range  of  BPI  potentials,  with  each  PIP  offering  some  advantage  over  the 
current  process,  and  with  each  PIP  implementation  building  upon  the  previous  PIP 
implementation  that  would  enable  ALCs  to  move  toward  or  achieve  their  targets.  Table  3-1 
summarizes  these  PIPs. 

The  PIPs  offer  a  range  of  improvement,  and  each  PEP  offers  some  advantage  over  the  PDM 
process  as  it  currently  exists.  However,  the  impact  should  be  measured  against  the  objectives  of 
substantially  reducing  organic  aircraft  PDM  operating  expense  and  flow  days.  PIP  D  offers  the 
greatest  ability  to  achieve  those  objectives.  PIP  A  offers  the  lesser  ability  to  achieve  those 

objectives. 

To  provide  a  basis  for  selecting  the  criteria  used  to  measure  the  cost  improvement  provided  by 
the  PIPs  within  the  business  case,  a  memorandum  dated  September  15, 1994  by  the  Secretary  of 
Defense  was  referenced.  In  this  memorandum  the  Secretary  challenged  the  Army,  Navy,  Air 
Force,  Marine  Corps,  and  the  Defense  Logistics  Agency  to  reduce  the  business  process  cycle 
times  by  at  least  50%  by  the  year  2000.  Using  this  target,  ^  well  as  information  and  ide^ 
provided  by  ALC  personnel  and  PDM  customers,  two  objectives  of  reducing  organic  aircraft 
PDM  operating  expenses  and  flow  days  were  developed  as  the  measurement  criteria  for  the  ITI- 
ALC  program  business  case. 
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Table  3-1.  PIP  Summary 


pjp  1  Description 

PIP  A 

(Process  Improvements  Only,  No  ITI-ALC  Technology) 

Implements  a  slice  from  BPIs  within  the  current  “AS-IS” 
paradigm  that  can  begin  now,  demonstote  the  new 
activity,  and  is  more  effective  and  efficient,  though  it 
does  not  include  ITI-ALC  system  technology. 

PIPE 

(Introductory  System) 

Implements  a  slice  from  BPIs  within  the  current  “AS-IS” 
paradigm  that  produces  more  improvements  in 
effectiveness  and  efficiency  than  PIP  A.  Some  ITI-ALC 
system  technology  is  introduced  to  provide  on-line 
access  to  individual  databases  and  a  single  interface  to 
core  depot  maintenance  systems.  This  PIP  does  not 
integrate  data.  Technical  data  is  not  organized  in  lETM 
format.  This  PIP  can  move  the  PDM  process  closest  to 
the  performance  targets  without  requiring  policy  changes 
outside  of  the  maintenance  process. 

PIPC 

(Integrated  Data) 

Implements  a  slice  from  BPIs  that  provides  integrated 
information,  introduces  lETM  data,  incorporates  more 
portable  ITI-ALC  system  technology,  establishes  the 
infrastructure  for  Organizational  level  (0-level)  to  Depot 
level  (D-level)  information  sharing,  and  allows  a  major 
breakout  of  the  current  process  and  a  major 
breakthrough  in  the  way  the  customer  is  served.  Saves 
more  by  enabling  the  reallocation  of  resources  to  the 
direct  maintenance  effort.  This  PIP  requires  a  paradigm 
Stretch. 

PIPD 

(Fully  Developed  ITI-ALC  System) 

Implements  a  slice  from  BPIs  that  incorporates  full  ITI- 
ALC  system  technology,  integrates  all  the  required 
systems  for  depot  maintenance  functionality,  provides 
artificially  intelligent  tools  to  support  all  the  BPIs, 
produces  information  as  a  by-product  of  the  work  effort, 
and  enables  the  final  step  toward  achieving  the  objec¬ 
tives.  This  PIP  will  require  a  major  paradigm  shift. 

For  each  PIP,  the  investment  cost  and  potential  benefit  for  these  two  criteria  were  evduated  from 
two  perspectives  -  that  of  the  maintenance  cost  to  the  depot  and  Aat  of  the  cost  to  the  customer. 
One  of  the  most  important  aspects  of  this  development  was  selecting  the  site  for  the  evaluation. 


The  site  selection  process  was  aimed  at  selecting  the  most  desirable  ALC  to  be  used  as  the  b 
for  developing  the  business  case  and  for  demonstrating  the  ITI-ALC  capability  dunng  Phase  I 
the  ITI-ALC  program.  The  final  recommendation  for  site  selection  will  be  made  jomtly  ^  the 
Government  program  manager  (AI7HRGO),  functional/domain  experts,  and  system/software 


engineers. 
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The  site  selection  process  was  accomplished  through  the  application  of  the  Analytic  Hierarchy 
Process  (AHP)  methodology,  based  on  work  done  by  Herbert  Simon.  The  steps  in  that 
methodology  were  applied  using  a  tool  called  Expert  Choice.  The  steps  were  as  follows. 

1 .  Problem  Definition  and  Research 

2.  Elimination  of  unfeasible  Alternatives  (low  pass  filter) 

3.  Evaluate  Candidates 

•  Determine  Selection  Criteria 

•  Prioritize/Weight  Criteria 

•  Make  Comparisons  (score  alternatives) 

•  Synthesize  Judgment  (Expert  Choice) 

•  Examine  &  Verify  Results  (sensitivity  analysis,  etc.) 

4.  Document  the  Decision 

These  steps  were  conducted  by  members  of  the  ITI-ALC  team  along  with  individuals  from 
HRGO.  A  draft  set  of  criteria  was  developed,  and  reviewed  and  ranked  by  the  team  members. 
The  individual  team  members  used  the  criteria  to  rank  each  of  the  ALCs.  These  rankings  were 
entered  into  the  EC  tool  to  provide  a  preliminary  look  at  the  results.  Using  the  results  of  the 
previous  steps,  two  workshops  were  held  during  which  the  previous  results  were  presented, 
discussed,  and  adjusted  to  obtain  a  team  consensus. 

Figure  3-4  shows  the  results  of  that  effort.  The  graph  indicates  the  score  of  each  of  the  ALCs. 
WR-ALC  and  SM-ALC  scores  indicate  no  significant  difference  using  the  criteria  and  approach 
outlined  above.  SM-ALC  was  identified  by  the  DoD  as  a  “privitization”  location  subsequent  to 
the  recommendations  of  the  Commission  as  Base  Closure  and  Realignment.  As  a  result,  the 
recommended  selected  site  for  the  demonstration  of  the  ITI-ALC  system  was  WR-ALC.  The  BC 
includes  benefit-cost  analysis  for  both  of  these  sites. 


Figure  3-4.  Results  of  the  Site  Selection  Process 
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3.4.2  BC  Validation 

An  operations  research  technique  was  used  to  test  the  validity  of  the  cost  and  benefit  estunabon 
results  of  the  FEA.  The  technique  used  was  a  “two-way-blind  test.”  The  intent  was  to  have  two 
separate  teams,  with  no  association/interaction,  and  using  two  different  approaches,  denve  results 
that  could  then  be  compared.  Ideally,  if  the  results  from  the  two  different  approaches  were 
similar,  this  would  indicate  the  results  represented  a  “real-world”  view  of  the  PDM  ^d  could  be 
considered  valid.  Also,  if  there  was  a  “small”  delta  between  the  two  results,  the  delta  became  a 
good  way  to  define  the  confidence  interval  for  the  results  (although  this  approach  to  defimng  the 
confidence  interval  obtains  only  a  locally  optimized  outcome).  The  two  approaches  were 
Engineering  Assessments  and  Simulations.  Very  early  on  in  the  project,  it  was  apparent  that  we 
did  not  have  the  resources  to  conduct  a  full  two-way  blind  test,  so  some  key  members  such  as  a 
cost  analyst  and  a  subject  matter  expert  were  on  both  teams.  This  provided  a  measure  of  conbo 
without  having  a  full  control  group  (a  third  redundant  team).  Refer  to  Section  4.4.4,  Possib  e 
Recommendations  and  Lessons  Learned  for  suggested  solution  to  this  problem. 

Furthermore,  because  simulations  are  a  repeatable  experiments,  “real-world”  historical  data 
could  be  input  into  the  simulations  and  compared  to  “real-world”  histoncal  results  to  c^ibrate 
the  simulations.  Then  the  simulation  engines  were  used  to  interpolate  results  given  the  ch^ges 
identified  in  the  BPIs  and  PIPs.  This  along  with  two-way-blind  testing  allows  greater  confidence 

in  the  results. 


3.4.2.1  Engineering  Assessments 

The  next  step  in  the  analysis  of  the  depot  maintenance  processes  represented  in  the  static  models 
was  to  perform  engineering  assessments.  During  these  engineering  assessments,  tae  ITI-^L 
team  applied  expert  judgment  to  the  processes  and  information  relationships  to  identify  potentials 
for  improvement.  The  depot  maintenance  processes  were  analyzed  using  the  following 

techniques: 


1 .  Initially  focusing  on  activities  with  the  greatest  resource  consumption. 

2.  Identifying  unnecessary  administrative  tasks,  approvals,  and  paperwork  for  removal. 

3.  Identifying,  for  removal,  identical  activities  performed  at  different  parts  of  the  process. 

4.  Evaluating  every  significant  activity  in  the  process  to  determine  its  contribution  to 
meeting  combat  command  requirements. 

5.  Reducing  the  complexity  of  the  process,  including  organizational  communication. 

6.  Identifying  ways  to  compress  cycle  time  to  meet  or  exceed  customer  expectations  and 
minimize  storage  costs. 

7.  Identifying  ways  to  facilitate  the  performance  of  activities. 

8.  Identifying  ways  to  more  effectively  use  capital  equipment  and  the  working  environment. 

9.  Identifying  single  ways  to  perform  an  activity  so  all  employees  always  do  the  activity  the 
same  way. 
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10.  Identifying  areas  where  the  quality  of  inputs  can  be  leveraged  to  improve  the  quality  of 
the  outputs. 

11.  Applying  tools,  equipment,  and  computers  to  routine  activities  to  free  up  employees  to 
accomplish  more  creative  activities. 


In  addition,  the  team  did  the  following: 

1  Applied  recommendations  and/or  lessons  learned  from  reports  of  pre^ous  ^d 

process  improvement  activities  in  DoD  and  other  federal  government  agencies  (refer  to 

Appendix  B  for  summaries  of  these  reports). 

2.  Collected  and  recorded  process  improvement  recommendations  from  mechanics  and 
other  ALC  personnel. 


3.  AppUed  best  practices  that  were  identified  during  visits  to  commercial  organizations 
performing  similar  maintenance  activities. 

4.  Performed  benefit/cost  analysis  with  business  case  analysts,  functional  experts  and 
information  engineers. 


3.4.2.2  Simulations 

The  results  of  the  data  collection,  modeling  efforts  and  engineering  assessment  were  tested 
SL  d^elmlation.  Various  models  provided  the  ftameworit  for  rite  stutdatton  (see 
Figure  3-5)  along  with  the  performance  data  collected  from  the  ALCs.  The  mode  mg  , 
Pr^iL  ^d  a  simulation  prLct,  WITNESS,  were  used  to  support  the  simulation  objectives 
Using  these  tools,  timing  constraints  and  resources  for  depot  mamtenance  operations  w 
defined  Characteristics  of  the  individual  processes  were  defined  with  of  probability  distributions 
tnmnriaS^&nS  m^^nan^  environment.  The  conditional  behavior  of  the  system  w^ 
STS  alset  th^^owTtes,  bottlenecks,  idle  time,  throughput,  cycle  times  worldoad,  and 
othtr  dynamic  properties.  Recognizing  the  potential  incompleteness  of  collected  data,  simulahon 
s^noriS^^m  analyses  to  define  performance  boundaries.  Potential  busmess  process 
improvements  were  simulated  first,  then  slices  of  major  process  improvemente  were 
proposals  The  result  was  a  series  of  process  improvement  recommendaUons.  Using  Aese 
^commendations,  the  ITI-ALC  team  developed  proposals  to  structure  viable  approaches 

achieving  the  objectives. 


Throughout  the  iterative  process,  Roupings  of  process  improvements  were  tested  using  SRA’s 
TurboBPR2  and  functional  economic  analysis  modeling  tools. 
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Figure  3-5.  Process  Model  (IDEF3)  Support  ofTTI-ALC 

3.43  BC  TraceabUity 

Information  extiacted  from  the  engineering  assessments,  process  model  efforte  and  stations 
was  used  to  help  refine  the  ITI-ALC  “TO-BE”  Functional  Model.  In  turn,  the  ITI-ALC  TO  BE 
Functional  Model  supported  the  ITI-ALC  system  requirements  documented  m  the  m-ALL 
System/Segment  Specification  (SRA,  October  1995)  and  the  business  reengineermg  concepts 
described  in  the  Business  Case.  Each  of  these  steps  to  the  process  had  a  formal  traceability  effort 
as  illustrated  in  Figure  3-3. 

3.4.4  BC  Possible  Recommendations  and  Lessons  Learned 

During  the  development  of  the  business  case,  the  following  were  possible  recommendations  or 
lessons  learned  that  were  identified. 


•  Domain  objective  were  not  available. 

There  were  no  AFMC-wide  objectives  dealing  explicitly  with  the  ITI-ALC  domain.  As  a 
result,  the  ITI-ALC  team  inferred  two  specific  priority  objectives  from  the  rnany  general 
objectives  included  in  AFMC  and  ALC  planning  documents.  Those  specific  objectives  were 
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reduced  operating  expense  and  aircraft  flow  days  for  organic  aircraft  PDM,  as  discussed  in 
the  business  case. 

Also,  there  were  no  command-wide  performance  measures  and  targets  (metrics)  dealing  with 
the  domain  that  ITI-ALC  was  intended  to  affect  Accordingly  the  team  inferred  measures  and 
targets  from  correspondence  and  data  available  within  the  depot  maintenance  domain. 

•  Process  improvement  proposal  structure  required  more  than  one  improvement 
alterative. 

This  project  team  recognized  that  previous  Information  Technology  (IT)  investment  projects 
typically  presented  decision  makers  with  one  alternative  for  unprovement  investment 
decisions.  Therefore  early  on  the  ITI-ALC  project  incorporated  an  initiative  to  present  a 
series  of  more  than  one  option  for  the  decision-maker.  Each  option  more  aggressive  than  the 
previous  option  in  the  series. 

•  Two-way-blind  test  could  not  be  fully  implemented. 

This  project  team  recognized  early  in  the  program  that  there  were  not  enough  “equal” 
resources  to  conduct  a  true  two-way  blind  test.  One  solution  to  this  issue  is  to  estimate 
enough  resources  in  the  project  cost  estimate.  This  is  very  expensive  and  may  not  be  possible 
if  equal  team  members  cannot  be  found.  If  equal  team  members  cannot  be  found  a  control 
team  would  also  have  to  be  formed  to  allow  for  the  evaluation  of  variables  between  the  two 
teams.  In  this  age  of  “do  more  with  less”,  this  is  not  an  acceptable  approach.  Using  two  core 
teams  with  one  or  two  team  members  in  both  teams  worked  as  long  as  everyone  excepts  the 
condition  that  the  learning  effect  may  bias  the  results.  To  minimize  this  bias,  different  team 
leaders  were  used  and  the  SME  was  kept  as  “blind  as  possible  to  the  overall  process  so  as 
not  to  subconsciously  direct  the  two  processes  so  they  artificially  converge.  TMs  plus  an 
independent  validation  on  the  simulation  half  of  the  experiment  (see  the  validation  section 
above),  allowed  for  results  with  small  confidence  intervals. 

•  Another  measure  for  economic  analysis. 

DoD  and  Air  Force  directives  on  economic  analysis  and  the  development  of  business  cases 
contain  two  views  of  economic  analysis. 

In  the  m-ALC  project,  the  traditional  view,  defines  the  problem  with  the  question,  “what  is 
the  maintenance  cost  to  accomplish  organic  aircraft  PDM  and  how  can  we  do  it  for  less?” 
The  answer  obviously  includes  the  cost  of  mechanics,  engineers,  and  managers,  the  parts  and 
raw  materials  incoiporated  into  the  aircraft  product,  the  hangers  and  other  facilities  and  the 
cost  of  supporting  resources.  It  might  be  termed  the  in-house  cost.  It  is  not  the  major  cost. 
This  view  deals  with  the  problem  from  a  systems  perspective.  It  does  not  call  into  question 
the  “out-of-service”  costs  associated  with  the  relationship  between  the  customer  and  the 
provider,  AFMC.  This  problem  is  not  new.  It  was  raised  by  the  DUSD(L)  in  1995. 
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In  a  May  30,  1995  memo,  the  DUSD(L)  made  a  statement  in  response  to  GAO/AIMD-95-1 10 
Depot  Maintenance  Standard  Systems.  The  statement  included,  “While  the  Department 
appreciates  the  GAO  acknowledgment  of  process  improvements  achieved  to  date  in  depot 
maintenance,  the  GAO  fails  to  realize  the  magnitude  of  achieving  significant  reductions  in 
cost  and  flow  days  equating  to  a  30%  reduction  in  the  cost  of  ship  overhaul,  or  processing 
two  additional  B-1  bombers  through  an  ALC  because  of  reductions  in  flow  days.  To 
illustrate,  due  to  the  complete  reengineering  of  work  processes  and  use  of  BAIM,  the  Navy 
hflg  reduced  overhaul  time  for  the  688  class  submarine  firom  24  to  20  months  and  now 
estimates  all  future  688  workloads  will  take  18  months.  To  put  that  change  in  perspective, 
the  previous  average  for  similar  work  was  $81  million  per  submarine.  However,  in  addition 
to  reducing  the  cost  of  overhaul,  weapon  systems  are  expedited  to  the  warfighter,  increasing 
mission  readiness.  In  addition  fewer  systems  in  the  repair  cycle  equates  to  fewer  systems 
needing  overall,  thereby  achieving  even  more  dramatic  savings.” 

Without  a  value  on  the  amount  of  time  a  system  is  out-of-service,  the  repair  cycle  days,  it  is 
difficult  for  managers  to  make  investments  to  reduce  them.  The  ITI-ALC  system  therefore 
developed  a  second  view  that  combines  total  process  and  cost  views  into  a  total  systems 
view.  It  includes  the  traditional  view  and  takes  into  account  the  needs  of  and  costs  to  the 
customer.  In  this  view,  the  question  is  not  only  “what  is  the  maintenance  costT;  but  in 
addition,  “what  are  the  costs  to  the  customer!”. 

What  does  the  customer  give  up  to  obtain  a  PDM  or  incorporation  of  a  modification  package 
in  an  aircraft?  A  customer  gives  up  two  things.  First,  a  customer  pays  the  maintenance  cost 
for  each  aircraft.  That  cost  is  relatively  straightforward  as  reflected  in  the  traditional  view 
discussed  above.  Second,  a  customer  relinquishes  use  of  that  portion  of  the  aircraft’s  life 
spends  in  the  maintenance  process.  For  PDM  the  period  will  vary,  but  ranges  fi'om  100  to 
several  hundreds  of  days  per  PDM  cycle. 

The  cost  to  the  customer  of  those  days  is  less  straightforward  than  the  representing  the 
maintenance  cost.  However,  it  was  developed  by  the  ITI-ALC  team,  based  on  similar 
industry  practice. 

To  develop  the  cost  to  the  customer,  the  ITI-ALC  team  obtained  the  Unit  Flyaway  Cost 
(UFC)  for  major  aircraft  in  the  USAF  fleet  from  Air  Force  Instruction  65-503.  This 
information  for  ten  aircraft  types  is  included  in  the  table  below.  These  items  are  included  in 
the  determination  of  a  unit  flyaway  cost  under  Appropriation  3010  (Aircraft  Procurement); 
airframe,  propulsion,  electronics,  avionics,  engineering  change  orders,  if  any,  government 
furnished  equipment,  first  destination  transportation  imless  a  separate  line  item,  system  and 
project  management  and  system  test  and  evaluation  if  funded  by  the  aircraft  procurement 
appropriation,  warranties,  recurring  costs  (both  contract  and  in-house),  nonrecurrmg  cost 
(both  contract  and  in-house),  and  advances  buy  cost  (see  Table  3-2). 
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Table  3-2.  Aircraft  Cost  Inforntation 


Aircraft  MDS 

AFI 65-503  UFC 
(millions  $) 

UFC  *  1.2 
(millions  $) 

Value  of  One 
Aircraft  Day  ($)^ 

B-1 

S6 

240.7 

288.8 

39566 

584 

24.2 

29 

3982 

1588 

14.5 

17.4 

2384 

F-22 

442 

68.1 

81.7 

11197 

E-3 

29 

114.3 

137.2 

18795 

C-5 

76 

135.6 

162.7 

22300 

C-130 

877 

14.5 

17.4 

2387 

KC-135R 

400 

52.7 

63.2 

8664 

C-141 

223 

41.2 

49.5 

6785 

C-17 

120 

293.2 

351.8 

48199 

'USAF  Fact  Sheets 

*FY95  to  FY94  conversion  aircraft  procurement  weighted  factor  of  1.032. 


Unit  flyaway  cost  does  not  include:  research,  test,  and  evaluation  appropriation  expenditures, 
weapons  and  armament  (except  if  part  of  the  airframe,  e.g.,  the  30MM  GAU-81 A  gun  on  the  A- 
10),  peculiar  ground  support  equipment,  peculiar  training  equipment,  technical  data,  initial  spares 

and  replacement  spares. 

In  regards  to  flyaway  cost  and  modifications,  it  is  important  to  note  that  UFC  reflects  only  those 
modifications  which  produced  a  new  MDS.  For  example,  the  EF-1 1 1 A  was  modified  from  the 
F-1 1 1  A.  Major  aircraft  modifications  which  do  not  produce  a  new  MDS  are  not  included.  Thus, 
the  unit  flyaway  cost  for  the  B-052H  reflects  the  unit  flyaway  cost  as  originally  produced  and 
then  inflated  to  the  constant  dollars  of  a  specific  fiscal  year.  Since  subsequent  modifications  to 
the  B-052H  did  not  produce  a  new  MDS,  the  modifications  are  not  included  in  the  unit  flyaway 
cost  of  the  B-052H. 

To  account  for  research,  development,  test  and  evaluation,  technical  data,  support  equipment, 
etc.,  the  m-ALC  team  increased  UFC  by  20%.  The  ITI-ALC  team  assumed  20  years  in  an 
aircraft  life  cycle. 

To  calculate  the  value  of  one  aircraft  flow  day  the  team  applied  this  formula. 

(UFC)(I.2) 

Cost  to  the  Customerfor  an  Aircraft  Flow  Day  -  ^ycle  in  Years) (365  Days) 

Using  this  approach,  the  value  of  MSIP  F-1 5  flow  days  of  174  days,  is  $692,868  per  aircraft. 
The  value  of  the  154  F-15  flow  days  provided  to  a  customer,  if  flow  days  could  be  reduced  to  20 
days,  is  $613,226  per  aircraft.  The  ITI-ALC  team  estimated  Primary  Aircraft  Authorization 
(PAA)  fleet  sizes  from  USAF  fact  sheets  or  public  literature.  For  a  fleet  of  584  F-15  aircraft,  the 
value  of  the  returned  flow  days  is  $358,125,152  over  one  5  or  6  year  PDM  cycle. 
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3.5  SYSTEM/SEGMENT  SPECIFICATION  (SSS)  CDRL  SEQUENCE  A008 

The  SSS  documents  the  requirements  for  the  ITI-ALC  system.  This  system  will  encompass 
activities,  processes,  and  information  used  to  support  mechanics,  planners,  controllers,  and 
managers  involved  with  PDM  on  aircraft  and  aircraft  components. 

3.5.1  SSS  Development 

The  ITI-ALC  SSS  was  created  through  an  iterative  development  process  begmmng  with  an 
examination  of  the  Data  Item  Description  (DID)  for  a  System/Segment  Specification  to 
determine  the  content  for  the  SSS.  This  information  was  roughly  allocated  to  the  specific 
deliveries  to  establish  customer  expectations  for  the  document  as  well  as  development  priorities. 
An  outline  of  the  SSS  was  created  with  brief  descriptions  of  the  expected  content  of  each  section 
and  an  approach  for  development  of  the  section  or  a  potential  source  of  the  data  for  the  section. 
Figure  3-6  depicts  the  major  components  of  the  SSS.  The  bulk  of  the  SSS  is  devoted  to 
requirements  which  fall  into  three  basic  categories:  user/fimctional  requirements,  external 
interface  requirements,  and  operational  requirements  (including  segment  requirements). 

The  primary  focus  for  the  SSS  was  determined  to  be  the  user/fimctional  requirements  (specified 
in  user  terminology)  which  would  then  drive  the  subsequent  sections  of  the  document. 
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Figure  3-6.  SSS  Components 
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3.5.1.1  Development  of  Capabilities 

The  hierarchy  to  be  used  within  the  user  requirements  section  was  roughly  defined  through 
analysis  of  the  UI-ALC  “TO-BE”  Functional  Model.  First,  the  major  capabilities  were  derived 
from  the  functional  model,  and  nodes  from  the  model  allocated  to  the  capabilities  accordingly.  A 
description  of  each  capability  was  created  to  identify  the  intent  of  each  capability  and  to  provide 
the  foundation  for  further  development.  This  step  was  followed  by  an  informal  review  of  the 
capabilities  and  node  allocation  with  the  ITI-ALC  project  manager  and  the  ITT-ALC  system 
modeler  to  validate  the  structure  and  approach,  as  well  as  ensuring  consistency  with  the  system 
model. 

3.5.1.2  Development  of  Functions 

Once  the  capabilities  were  relatively  stable,  the  flmctional  model  nodes  allocated  to  each 
capability  were  analyzed  in  order  to  define  the  next  level  of  breakout  functions.  Functions 
were  defined  within  each  capability  and  the  nodes  allocated  to  each  function  accordingly. 
Descriptions  were  written  for  each  function  to  identify  the  intent  of  the  function  and  to  provide 
guidance  for  the  next  level  of  development  for  each  function.  The  definition  of  functions  was 
followed  by  an  internal  review  with  the  previous  audience  (project  manager  and  system  modeler) 
and  the  addition  of  the  functional  modeler  and  the  functional  expert.  This  review  focused  on 
further  validation  of  the  capabilities,  examination  and  refinement  of  the  functions,  and  ensuring 
the  complete  coverage  of  appropriate  depot  maintenance  activities  in  the  planned  structure  for  the 
SSS. 

3.5.13  Development  of  Other  Requirements 

Following  the  definition  of  functions,  further  analysis  of  the  “TO-BE”  functional  model  was 
conducted  to  develop  the  requirements.  The  analysis  was  based  on  the  nodes  that  were  allocated 
to  a  given  function,  and  the  description  of  the  purpose  of  the  function  (which  implied  certain 
requirements).  The  focus  of  this  iteration  was  development  of  the  user  functional  requirements 
in  order  to  support  the  SSS  Development  Workshop  (refer  to  Section  4.5. 1.2).  Other 
requirements  such  as  the  segment  requirements,  the  external  interface  requirements,  and  the 
operational  requirements  were  addressed  after  the  user  requirements  were  relatively  stable  and 
mature.  All  requirements  were  developed  as  succinct,  concise,  one-sentence  statements 
reflecting  what  the  system  needed  to  do.  The  “TO-BE”  functional  model  node  number  from 
which  the  requirement  was  derived  was  appended  to  the  end  of  each  requirement  to  support 
requirement  tracing.  Each  user  requirement  was  also  accompanied  by  a  small  paragraph  labeled 
“DISCUSSION”  that  provided  additional  detail  for  imderstanding  the  associated  requirement  as 
well  as  possible  implementation  considerations.  Figure  3-7  depicts  the  organization  of  the  SSS 
functional  requirements. 
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Figure  3-7.  Functional  Requirements  Organization 

3.5.1.4  External  Interface  Requirements  Development 

The  external  interface  requirements  were  developed  to  be  consistent  with  and  supportive  of  the 
user  requirements.  The  external  interface  requirements  were  derived  from  &e  relationsbps  to 
external  systems  as  reflected  in  the  ITI-ALC  Architecture  Report  particul^ly  the  TO-BE 
Functional  Model  and  the  System  Model.  These  requirements  reflect  the  ^a  to  be 

passed  between  IThALC  and  each  identified  system  at  a  high  level.  Followmg  delivery  of  Ae 
preliminary  final  SSS,  an  internal  meeting  was  held  to 

information  reflected  for  each  external  interface  with  the  information  ^ntamed  in  Ae  System 
Model.  The  attendees  for  this  meeting  were  the  system  modeler,  the  SSDD  author,  the  project 
functional  expert,  and  the  SSS  author. 


3.5.1.5  Operational  Requirements  Development 

The  operational  requirements  include  the  requirements  for  system  quality,  security,  reliability  and 
maintainability,  and  documentation.  These  requirements  were  mcorporated  mto  the  preliminary 
final  of  the  SSS  and  were  basically  copied  from  the  IMIS  requirements  for  these  same  areas. 
Changes  were  made  as  appropriate  to  reflect  things  peculiar  to  the  ITI-ALC  system. 

The  requirements  for  the  segments  were  also  developed  as  part  of  &e  operational  requi^mente 
for  the  system.  Initially,  six  segments  were  defined  and  reviewed  internally  by  the  ITI- 
team.  Descriptions  were  written  for  each  segment  to  address  the  purpose  of  the  segment,  the 
primary  user  of  the  segment,  and  general  characteristics  for  the  segment.  It  was  determmed  tha 
some  of  the  operational  requirements  to  be  defined  according  to  DID  were  more  applicable  a 
segment  level.  Consequently  the  requirements  for  computer  resource  reserve  capacity,  physicm 
characteristics,  and  environmental  requirements  were  all  defined  at  the  segment  leve . 
Requirements  were  written  for  each  segment  addressing  each  of  these  areas.  These  requirements 
were  included  in  the  draft  version  of  the  SSS.  Comments  from  the  customer  indicated  the 
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proposed  segments  and  associated  requirements  were  premature  in  the  level  of  detail  provided. 
The  segments  were  redefined  as  three  more  generic  segments  fi-om  which  the  ori^al  six 
hardware  items  could  be  derived  representing  one  implementation  of  the  se^ents.  Figure  3-8 
depicts  the  revised  segments.  These  three  segments  along  with  a  more  generic  set  of  associated 
requirements  were  incorporated  into  the  final  SSS. 

The  representation  of  segments  within  the  SSS  was  revised  to  reflect  more  generic  classes  of 
items  with  associated  generic  requirements  instead  of  specific  devices.  The  original  six  hardware 
items  were  included  with  the  definition  of  the  three  segments  as  indicated  in  Figure  3-8  to 
provide  context  for  each  segment  as  a  representation  of  a  possible  implementation  of  the 

segments. 

The  six  hardware  items  were  further  defined  in  the  draft  version  of  the  SSDD.  Care  was  taken 
when  identifying  the  three  segments  to  ensure  that  the  six  hardware  items  in  the  SSDD  could  be 
derived  from  the  segments  as  a  possible  implementation.  The  basic  philosophy  for  the  segments 
is  that  multiple  devices  may  be  used  to  implement  a  given  segment,  but  it  is  not  imperative  that  a 
single  device  satisfy  all  the  requirements  for  the  segment.  However,  if  multiple  devices  are  used 
to  represent  a  segment,  those  devices  must  collectively  satisfy  all  the  requirements  for  that 

segment. 
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Figure  3-8.  ITI-ALC  Segment  Implementation  Concept 
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3.5.2  SSS  Validation 


Throughout  development  of  the  SSS,  multiple  internal  reviews  were  conducted,  each  with  more 
participants  than  the  previous  review  as  appropriate  to  ensure  the  integrity  of  the  review,  to  keep 
the  reviews  manageable,  and  to  focus  on  the  project  members  most  affected  by  the  information 
represented  in  the  document.  Figure  3-9  is  a  graphical  representation  of  the  process  used  to 

develop  the  ITI-ALC  SSS. 


PfUEUMNARYnNAL 


Figure  3-9.  SSS  Iterative  Development  Strategy 

Once  the  requirements  were  defined,  a  “strawman”  version  of  the  document  was  compiled  for 
use  in  the  SSS  Development  Workshop.  The  purpose  of  the  workshop  was  to  review,  revise,  and 
validate  the  proposed  ITI-ALC  system  (business  processes,  system  requirements,  systeni 
hardware)  as  represented  in  the  SSS.  The  output  of  the  workshop  was  a  revised  SSS  that  formed 
the  basis  for  the  second  delivery  (preliminary  final). 

The  workshop  had  participants  fi-om  four  ALCs:  00-ALC,  SM-ALC,  WR-ALC,  and  OC-ALC. 
The  intent  was  to  have  representation  from  all  five  ALCs,  but  due  to  a  miscommunication  e 
SA-ALC  was  not  represented.  The  backgrounds  of  the  ALC  participants  consisted  of  pliers, 
managers,  mechanics,  an  industrial  engineer,  and  F-22  personnel.  Additional  particip^on 
consisted  of  our  fimctional  expert,  fimctional  and  system  modelers,  representatives  from 
AL/HRGO,  and  SRA  project  management.  The  selection  of  participants  was  designed  to  obtain 
a  cross-fimctional  set  of  people  representing  the  aspects  of  depot  maintenance  that  have  been  our 
primary  focus. 
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The  workshop  was  held  for  3  days  beginning  with  an  overview  of  the  proposed  ITI-ALC  system 
and  a  review  of  the  capabilities  contained  in  the  SSS  with  a  brief  discussion  of  the  functions 
associated  with  each  capability.  Participants  were  divided  into  three  teams  responsible  for 
completing  the  group  exercises  and  presenting  the  work  accomplished  by  each  team.  The  re^ts 
of  these  group  exercises  were  revisions  to  the  descriptions  of  the  capabilities  and  functions, 
revisions  to  the  requirements,  and  miscellaneous  changes  including  deletion  of  several 
requirements  and  one  function. 

Following  the  workshop,  the  changes  were  incorporated  into  the  preliminary  final  version  of  the 
SSS.  A  report  identifying  the  changes  to  the  SSS  resulting  from  the  workshop  was  developed 
and  provided  to  the  ALC  participants. 

3.5.3  SSS  Traceability 

Like  quality  assurance,  traceability  should  be  accomplished  throughout  the  process  of  development 
as  well  as  a  final  verification  step  near  the  end  of  file  development  process.  The  ITI-ALC  team 
used  an  innovative  ^proach  by  developing  the  “structure”  of  our  process  so  that  traceability  was 
done  and  not  as  a  “second  step.”  As  can  be  read  in  the  sections  above,  each 

requirement  was  developed  almost  directly  from  the  description  of  a  given  node  within  the  “TO- 
BE”  Functional  Model.  This  allowed  for  automatic  trace  to  that  node  as  well  as  the  indirect  trace 
back  to  at  least  one  user  interview  and  to  die  entities  in  the  TO-BE  Data  Model.  To  ensure  this 
traceability,  a  formal  trace  step  was  also  performed. 

Formal  traceability  between  the  ITI-ALC  Architecture  Report  and  the  ITI-ALC  SSS  consisted  of 
tracing  the  system  functions  and  functional  interfaces  within  the  SSS  to  the  activity  nodes  of  the 
“TO-BE”  models  in  the  architecture  report  from  which  they  were  derived  (see  Figure  3-3).  Within 
the  SSS,  each  system  function  contained  a  set  of  individual  system  requirements  which  were 
separately  identified  and  managed  to  facilitate  traceability.  As  a  result,  each  requirement  within  a 
given  system  function  of  the  SSS  also  traced  to  the  “TO-BE”  model  activity  node  from  which  the 
function  was  derived.  Additionally,  traceability  from  the  ITI-ALC  SSS  to  the  IMIS  SSS  was 
established  was  ensure  that  the  ITI-ALC  architecture  complimented  the  IMIS  architecture. 

3.5.4  SSS  Possible  Recommendations  and  Lessons  Learned 

The  process  used  to  develop  the  ITI-ALC  SSS  was  rigorous,  systematic,  and  successful.  There 
were  several  lessons  learned  from  implementing  this  process.  Also,  some  possible 
recommendations  may  apply. 

•  SSS  development  workshop  should  be  longer. 

The  workshop  was  three  days  long  which  only  allowed  time  for  revising  the  user 
requirements.  The  workshop  should  be  five  days  long  to  allow  time  to  examine  and  revise  all 
requirements  in  the  SSS  with  focus  on  the  user  requirements,  external  interface  requirements, 
and  segment  descriptions  and  requirements. 
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•  SSS  workshop  instead  of  one  of  the  three  deliverables. 

If  a  workshop  is  used  as  a  means  to  develop  the  SSS,  there  should  only  be  two  deliveries  of 
the  document.  There  should  be  one  delivery  (preliminary  final)  reflecting  the  results  of  the 
workshop  and  a  second  delivery  (final)  approximately  three  months  later  to  address  any 
outstanding  details  or  issues  resulting  from  customer  review  of  the  post-workshop  version. 
The  premise  of  two  deliveries  stems  firom  the  idea  that  the  workshop  is  supported  by  the 
appropriate  personnel  that  can  and  should  influence  the  requirements  within  the  SSS, 
negating  the  need  for  three  deliveries  to  achieve  the  desired  content  level. 

•  Iterative  requirements  development  is  required. 

The  primary  categories  of  developmental  requirements  within  the  SSS  are  functional 
requirements  (representing  user  requirements),  external  interface  requirements,  and  segment 
requirements.  The  functional  requirements  should  be  defined  first  and  should  determine  the 
need  and  provide  the  basis  for  the  external  interface  requirements  and  the  segment 
requirements.  As  requirements  are  developed,  each  category  (functional,  external  interface, 
and  segment)  should  be  revisited  and  revised  accordingly  (iterative  process)  until  “all” 
requirements  have  been  defined  for  the  system. 

•  The  concept  for  how  to  specify  and  organize  an  effective  SSS  was  not  consistent  among 
all  individuals  involved  in  its  development,  even  though  a  DID  was  identified  within  the 
contract. 

The  SSS  was  one  of  the  key  documents  developed  during  the  ITI-ALC  program  in  that  it 
specified  the  detailed  requirements  for  the  "TO-BE"  ITI-ALC  process  as  well  as  the 
technologies  used  to  implement  the  process.  The  true  success  of  the  ITI-ALC  program  will 
not  be  determined  until  the  SSS  is  used  as  the  guide  for  designing,  developing,  and 
implementing  the  ITI-ALC  system.  Given  the  importance  of  the  SSS,  the  “correct”  DID,  and 
a  full  understanding  of  that  DID,  should  have  been  established  early-on  by  all  representatives 
of  the  customer  and  should  have  been  reinforced  throughout  the  program.  The  reinforcement 
would  ensure  that  new  program  personnel  understand  the  history  and  goals  of  the  work 
performed  and  would  eliminate  the  possibility  of  unnecessary  program  direction  changes. 
These  variations  in  SSS  expectations  and  concepts  within  the  customer  team  required 
significant  time  and  effort  to  discuss  and  a  large  amount  of  rework. 

3.6  SYSTEMS  DESIGN  AND  ANALYSIS 

3.6.1  System/Segment  Design  Document  (SSDD)  CDRL  Sequence  A014 

The  ITI-ALC  SSDD  was  created  through  an  iterative  development  process  beginning  with  an 
examination  of  the  DID  for  the  SSDD  to  determine  the  content  of  the  SSDD.  This  information 
was  roughly  allocated  to  the  specific  deliveries  to  establish  customer  expectations  for  Ae 
document  as  well  as  development  priorities.  An  outline  of  the  SSDD  was  created  with  brief 
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descriptions  of  the  expected  content  of  each  section  and  an  approach  for  developm<Mt  of  fte 
section  or  a  potential  source  of  the  data  for  the  section.  Fi^  3-10  depicts  the  nmjor 
components  of  the  SSDD  and  where  information  that  supported  each  component  can  be  found. 
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Figure  3-10.  Information  Flow  to  SSDD 

3.6.1.1  SSDD  Development 

Tbe  system  development  process  for  phase  one  of  the  ITI-ALC  project  involved  devdopmg 

functional,  data,  and  process  models.  Also  developed  ^ultSe 

culminating  with  the  SSDD.  Throughout  the  development  of  each  release  of  the  S^D  multiple 
internal  and  customer  reviews  were  conducted  to  help  ensure  the  mtegrity  of  the  doc^ent. 
Internal  reviews  were  conducted  by  functional  experts  and  project  model  and  document  authors. 

The  m-ALC  proiect’s  approach  was  to  work  on  the  specific  sections  of  the  SSDD  after 
completion  of  Lsociated  supporting  sections  of  the  SSS  and  system  model  documents  we 
through  at  least  their  first  iteration  and  reviewed  by  the  customer.  As  later  iterati^s  of  Ae 
documents  were  reviewed  new  information  was  incorporate  mto  the  next  version  of  the  SSDD. 
This  allows  for  a  shorter  project  schedule  because  of  the  overlappmg  of  the  development 
schedule  of  each  document. 


56 


Once  the  appropriate  sections  of  the  SM  had  been  completed  and  reviewed,  the  resulting  model 
was  used  to  identify  portions  of  the  system  overview  as  well  as  the  software  portion  of  the  system 
hierarchy.  The  SM  also  helped  CSCI-to-CSCI  interfaces,  internal  CSCI  interfaces,  external 
interfaces,  and  inputs  and  outputs  of  each  process.  Figure  3-10  represents  how  the  SM  was  used 
to  directly  develop  major  components  of  the  SSDD. 

Also  shown  are  other  important  sources  of  information  for  the  SSDD,  such  as  the  SSS.  The  SSS 
was  instrumental  in  the  development  of  the  system’s  mission  and  especially  the  user’s  needs. 
The  SSS  defined,  in  a  concise  fashion,  the  specific  needs  a  system  would  have  to  fulfill  in  order 
to  satisfy  the  system’s  mission,  along  with  each  type  of  users  needs.  The  functions  defined  in 
the  SSS  along  with  the  users  interfaces  section,  and  the  SM  enabled  us  to  define  the  ^stem’s 
software  components  used  to  fulfill  functional  and  user  requirements.  The  External  interface 
requirements  section  of  the  SSS  was  also  used  to  help  determine  what  fractions  and  hardware 
would  be  needed  to  communicate  to  other  systems.  The  Operational  requirements  defined  in  the 
SSS  were  a  major  source  used  in  the  development  of  the  hardware  segments.  These 
requirements  spelled  out  the  purpose  of  each  segment,  user  of  the  segment  and  described  the 
general  characteristics  of  the  segment.  This  information  helped  define  the  look,  feel  and 
capabilities  of  the  final  segments. 

The  last  major  part  of  system  development  was  the  architecture  of  the  ITI-ALC  system.  The 
Corporate  Information  Management  (CIM)  Technical  Reference  Model  and  DoD’s  Technical 
Architecture  Framework  for  Information  Management  (TAFIM)  documentation  were  excellent 
starting  points  to  an  overall  system  architecture  for  ITI-ALC.  The  CIM  Technical  Reference 
Model,  populated  with  approved  standards,  was  used  as  a  major  source  of  information.  It  was 
determined  that  following  a  standard  would  ensure  the  resulting  system  would  be  developed 
using  solid  approved  and  proven  services  thus  avoiding  the  potential  pitfalls  of  yet  unproved 
services.  Furthermore,  the  resulting  system  would  be  more  open  to  existing  and  future  system 
built  under  the  same  premise. 

To  ensure  that  the  ITI-ALC  system  truly  meet  the  users  needed,  a  few  additions  were  added  to 
the  standard  services.  These  include:  C,  C++,  EETM-M,  D  and  Q  and  Commumcations  service. 
The  C  and  C++  programming  languages  were  added  to  help  better  create  the  user  interfaces  and 
system  communications.  With  these  languages,  a  large  amount  of  versatility  and  reuse  potential 
can  be  used  to  enhancing  the  overall  final  product.  The  lETM  specs  were  added  because  The 
ITI-ALC  system  can  be  characterized  as  an  lETM  based  system  and  the  CIM  model  didn  t 
contain  any  EETM  type  specifications.  The  communication  services  section  was  added  because 
the  ITI-ALC  system  will  have  to  communicate  with  other  systems  including  legacy  systems  that 
were  not  developed  using  open  system  concepts  and  therefore  will  not  conform  to  standards 
identified  in  the  network  services  of  CIM  Technical  Reference  Model  or  TAFIM. 

3.6.1.2  SSDD  Validation 

The  systematic  and  rigorous  process  used  to  develop  the  SSDD  is  shown  in  Figxire  3-11.  The 
development  process  is  based  on  information  and  analysis  of  other  models  and  documents  that 
have  undergone  many  reviews  conducted  by  fractional  experts,  modelers,  authors,  designers,  and 
developers  during  the  performance  of  the  ITI-ALC  program.  Both  the  SSS  requirements  and  SM 
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Figure  3-11.  Iterative  Development  of  a  System 

design  were  derived  from  requirements  established  in  the  functional,  process,  and  data  models. 
To  ensure  consistency  between  the  two  methods,  reviews  were  completed.  Both  documents  were 
essential  in  the  development  of  the  SSDD. 

This  structure  along  with  review  of  the  document  help  to  ensure  that  a  valid  design  was 
generated.  The  focus  of  the  SSDD  review  process  addressed  two  perspectives.  First  &e  reviews 
Lured  that:  (1)  correlation  exists  between  the  SSDD  and  the  “TO-BE”  versions  of  the  FM  ^d 
DM  (2)  the  SSDD  conforms  to  the  DID,  (3)  all  terms  are  fully  defined  m  the  glossa^,  and  (4) 
the  correlation  with  other  depot  maintenance  information  is  maximized.  Second,  the  reviews 
ensured  that  the  document  provided  an  understandable  representation  such  &at  an  eff^tive 
validation  process  can  be  performed.  Both  perspectives  involved  usmg  walkthroughs  and  peer 
reviews  by  internal  functional  experts  and  systems/software  engineers. 

One  peer  walkthrough  review  was  conducted  on  this  document  as  a  desk  top  review.  Peer  reviews 
provide  early  visMity  into  the  status  and  quality  of  the  product  with  ample  time  to  take  cojective 
action  if  progress  and  quality  are  not  at  acceptable  levels.  One  subject  matter  expert  walkthrough 
review  was  conducted  to  ensure  that  the  SSDD  was  usable,  as  well  as  coniprehensive  ^d 
compliant  with  project  needs.  Upon  completion  of  these  reviews,  the  review  leader  collected  the 
reviewer’s  comments,  evaluated  the  results,  and  defined  any  action  items. 

Next,  this  model  was  validated.  The  focus  of  the  validation  process  was  to  ensure  that  SSDD 

provided  a  complete  and  accurate  representation  of  the  ITI-ALC  system.  Because  the  TO-BE 
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FM  controlled  the  baseline  processes  that  must  be  supported  by  the  system  specified  by  the 
SSDD,  the  validation  of  the  FM  provided  a  significant  step  toward  the  validation  of  the  SSDD. 
The  validation  of  the  linkage  between  these  entities  was  facilitated  by  the  tracing  capability 
provided  by  the  RTC. 

3.6.13  SSDD  Traceability 

Since  the  SSS  encompassed  all  requirements  applicable  to  the  ITI-ALC  system  and  served  as  a 
basis  for  the  SSDD,  the  SSDD  need  only  trace  directly  back  to  the  SSS.  Traceability  between  the 
m-ALC  SSS  and  the  ITI-ALC  SSDD  consisted  of  tracing  the  system  functions,  requirements,  and 
functional  interfaces  of  the  SSS  to  the  Hardware  Configuration  Items  (HWCIs),  CSCIs,  and  manual 
operations  contained  in  the  SSDD.  For  SSS-to-SSDD  traceability,  each  requirement  within  the 
SSS  traced  to  the  components)  within  the  SSDD  that  satisfied  that  requirement  and  to  which  that 
requirement  was  allocated.  Some  SSS  requirements  were  allocated  to  either  a  HWCI  or  CSCI, 
some  were  allocated  across  multiple  HWCIs  and/or  CSCIs,  and  others  were  allocat^  to  manual 
operations.  SSS  requirements  were  written  to  facilitate  allocation  to  single  configuration  items  and 
to  minimiyf^  the  need  for  allocation  across  multiple  configuration  items. 

3.6.1.4  SSDD  Possible  Recommendations  and  Lessons  Learned 

The  following  are  possible  recommendations  and/or  lessons  learned  were  discovered  d\iring  the 
development  the  SSDD. 

•  Overlap/parallel  development  of  project  deliverables. 

While  overlapping  the  development  of  project  documents  such  as  the  SSDD  and  the  SSS 
helps  reduce  the  project’s  schedule,  this  overlap  also  can  cause  extra,  non-v^ue-added  work 
when  changes  are  made,  due  to  the  ripple  effect.  The  solution  used  on  this  project  didn’t 
eliminate  this  rework  completely,  but  did  reduce  it  to  tolerable  levels.  The  primary  way  this 
was  done  was  to  structure  the  development  approach  to  take  advantage  of  the  iterative  nature 
of  the  system  development  process  (see  Figure  3-11).  The  “TO-BE”  models  represented 
requirements  for  the  system  and  were  baselined  early  enough  in  the  process  to  be  used  to 
develop  the  SM.  The  SM  then  became  the  major  input  into  the  SSDD.  Because  both  the 
SSS  and  the  SSDD  were  ultimately  derived  fi-om  the  same  sources,  the  SSDD  folfilled  all  the 
requirements  identified  in  the  SSS  even  with  them  being  developed  almost  simultaneously. 
This  fact  was  validated  using  the  traceability  process  outlined  above.  This  iimoyative 
approach  to  parallel  development  allowed  shorter  schedule  time  as  well  as  flexibility  to 

change. 

3.6.2  m-ALC-to-IMIS  IPS  Demo  Analysis  -  CDRL  Sequence  AOll 

The  results  of  the  effort  outlined  below  have  all  been  documented  in  the  Depot  Requirements 
Compared  to  IMIS  Prototype  Capabilities  Final  Report  (CDRL  number  AOll). 

3.6.2.1  IMIS  Demo  Analysis  Development 

The  process  shown  in  Figure  3-12  illustrates  the  process  used  to  develop  the  comparison 
document. 
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Figure  3-12.  Comparison  Process 

The  ITI-ALC-to-IMIS  IPS  demo  analysis  was  accomplished  by  comparing  the  functional 
requirements  for  ITI-ALC  as  defined  in  the  SSS,  the  functionality  of  the  Lab  IMIS  as  defined  in 
the  Lab  IMIS  user’s  manual,  and  the  functionality  of  the  DOSAVindows  version  of  the  PS.  The 
functional  requirements  of  the  ITI-ALC  system  were  mapped  to  the  functions  of  the  Lab  IMIS 
and  the  PS.  The  mapping  process  consisted  of  a  desktop  review  of  the  documentation  for  the 
ITI-ALC  and  Lab  IMIS  systems  and  operating  the  PS  on  a  486  desktop  PC  with  16  MB  of  RAM 
DOS  6.2  and  Windows  3.1.  No  documentation  was  provided  for  the  PS;  therefore,  the 
mapping  was  based  on  titles  of  the  functions  and  data  fields  presented  within  each  of  the  fields. 


3.6.2.2  IMIS  Demo  Analysis  Validation 

The  results  of  the  comparison  are  included  in  the  comparison  document  as  a  matrix  which  is 
explained  in  Figure  3-13.  The  matrix  lists  the  functions  of  the  PS  and  Lab  IMIS  in  the  first  two 
rows.  The  ITI-ALC  functions  are  listed  in  the  column  along  the  left  side  of  the  matrix.  The 
highlighted  cells  in  the  “Lab  IMIS”  row  are  pS  functions  that  are  not  available  in  the  Lab  IMIS 
and  the  highlighted  cells  in  the  “PS  function”  row  are  Lab  IMIS  functions  not  supported  by  PS. 


Columns  that  have  horizontal  hatches  identify  the  PS  and  Lab  IMIS  functions  that  are  not 
required  by  ITI-ALC.  The  ITI-ALC  functions  that  have  the  entire  row  highlighted  ^e  not 
available  in  either  the  PS  or  Lab  IMIS.  The  rows  that  are  not  highlighted  are  the  functions  of 
m-ALC  that  are  a  part  of  the  PS  and  Lab  IMIS.  The  ITI-ALC  functions  are  further  identified 
by  coding  to  the  PS  and  to  the  Lab  IMIS.  The  code  on  top  of  the  cross  mark  (A  or  B)  defines 
how  the  ITI-ALC  function  relates  to  the  corresponding  PS  function.  The  code  below  the  cross 
mark  (1  or  2)  defines  how  the  ITI-ALC  function  relates  to  the  Lab  IMIS  function.The  definition 
of  the  comparison  codes  are: 

Category  A  —  This  category  includes  those  PS  functions  that  support  ITI-ALC.  Category  A 
includes  those  PS  functions  that  are  the  same  as  IH-ALC  with  different  names,  (e.g.,  work  order 
in  the  PS  is  the  same  as  asset  or  reparable  plan  for  ITI-ALC). 

Category  B  -  This  category  includes  those  PS  functions  that  could  support  ITI-ALC  but  are  not 
implemented. 
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Figure  3-13.  Comparison  Matrix  Structure 

Category  1  -  This  category  includes  those  Lab  IMIS  functions  that  map  to  identical  functions  of 
the  ITI-ALC  system.  The  category  1  functions  are  those  that  are  required  for  utilizing  the  IMIS 
technology  at  both  the  O-  and  D-levels  of  maintenance.  The  functionality  includes  both  the  PDM 
and  backshop  maintenance  activities  conducted  at  the  ALCs. 

Category  2  -  This  category  includes  those  Lab  IMIS  functions  that  are  mappable  to  comparable 
functions  of  ITI-ALC;  however,  the  Lab  IMIS  requires  additional  refinements  such  as  additional 
or  reduced  data  fields,  interfacing  with  different  legacy  systems,  a  capability  to  determine  which 
information  is  correct,  and  different  data  transfer  rules. 

In  all  but  three  instances  where  the  ITI-ALC  system  functions  map  to  IPS  and  Lab  IMIS,  there 
are  multiple  mappings  between  a  single  ITI-ALC  function  and  the  IPS  and  Lab  IMIS  function. 
The  reason  for  the  multiple  mappings  is  that  the  ITI-ALC  system  is  in  the  definition  stage  while 
the  PS  and  Lab  IMIS  are  defined  and  developed  systems.  Therefore,  the  ITI-ALC  system 
functions  are  at  a  higher  level  than  the  developed  systems  and  the  functions  of  PS  and  Lab  IMIS 
were  mapped  to  functions  that  were  encompassed  in  the  higher  level  ITI-ALC  system  functions. 
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3.6.23  IMIS  Demo  Analysis  Traceability 

Detail  mapping  of  the  ITI-ALC  system  functions  to  the  functions  of  the  IPS  and  Lab  IMIS  was 
performed  as  part  of  this  effort.  TTie  mapping  of  the  functions  was  performed  by  using  the 
function  descriptions  in  the  ITI-ALC  SSS  and  comparing  them  to  the  functionality  that  is 
described  in  the  Lab  IMIS  user’s  manual  and  the  functionality  performed  by  the  IPS  software. 

3.6.2.4  IMIS  Demo  Analysis  Possible  Recommendations  and  Lessons  Learned 

During  the  performance  of  the  IMIS  Demo  Analysis,  the  following  possible  recommendations 

and/or  lessons  were  identified. 

•  System  documentation  is  needed. 

Comparing  a  developed  system  with  a  system  undergoing  requirements  definition  presented  a 
unique  challenge  that  would  have  been  reduced  if  the  documentation  for  the  developed 
system  had  been  provided. 

•  Working  with  the  software  of  the  developed  system  was  very  useful  for  determining  the 
functionality  of  the  system. 

Reading  the  documentation  for  a  system  helps  rmderstand  the  intended  functionality  of  the 
system  and  provides  the  mechanics  necessary  to  maneuver  within  the  system.  Having  gained 
this  fundamental  imderstanding,  getting  hands-on  experience  is  the  only  way  of 
understanding  the  real  system  capabilities  necessary  to  perform  an  indepth  evaluation  and 
comparison. 

•  Development  system  source  code  would  have  enhanced  the  comparison. 

The  comparison  document  could  have  provided  another  level  of  usefulness  if  the  software 
code  had  been  provided.  The  code  would  have  allowed  the  identification  of  the  changes 
required  in  the  code  rather  than  the  functions  that  require  change. 

3.7  FINAL  DEMONSTRATION/EVALUTION  PLAN  -  CDRL  SEQUENCE  AGIO 

The  in-ALC  demonstration  effort  will  be  conducted  on  a  PDM  line  at  a  selected  ALC  site.  The 
objective  of  the  m-ALC  demonstration  is  to  compare  the  effectiveness  and  efficiency  of 
mechanics  and  other  depot  PDM  personnel  when  using  new  processes  and  technologies 
described  in  the  ITI-ALC  System/Segment  Specification  and  the  ITI-ALC  Business  Case.  To  do 
this,  the  planning  for  the  demonstration  was  done  like  a  formal  test  planning  or  experiment 
design  effort.  In  general,  all  formal  testing  is  done  with  one  of  four  techniques;  (1) 
demonstration,  (2)  test,  (3)  analysis,  or  (4)  inspection. 

The  ITI-ALC  Final  Demonstration/Evaluation  Plan,  resulted  from  this  effort  utilized  the  first 
three  of  these  techniques.  The  system/process  will  be  authenticated  employing  the  technique  of 
either  a  demonstration  or  a  test.  Analysis  will  be  used,  among  other  things  to  prove  hypotheses 
(e.g.,  using  the  simulations  created  during  the  first  phase  of  this  project)  that  cannot  be 
effectively  proven  using  any  other  technique.  The  intent  of  a  demonstration  will  be  to  show 
some  aspect  of  the  processes  or  technologies  that  could  be  implemented  in  a  production-level 
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ITI-ALC  system.  The  intent  of  a  test  is  to  obtain  empirical  data  so  that  formal  analysis  can  be 
conducted  on  a  new  process  or  technology  to  assess  performance.  In  all  ^es  performance  data 
from  individuals  using  the  new  methods  of  performing  a  given  tasks  will  be  compared  to  the 
performance  data  of  individuals  who  are  using  standard  (current)  methods. 

3.7.1  Demonstration/Evaluation  Plan  Development 

In  any  test  or  demonstration  being  conducted  like  an  experiment,  the  “experimenter  is 
attempting  to  draw  certain  inferences  or  make  a  decision  about  some  hypothesis  that  concerning 
the  situation  being  studied.  Experimental  designed  (which  this  effort  could  be  classified  as),  are 
used  to  organize  complicated  experiments  and  to  help  reduce  the  error  in  data  collecting. 
Randomization  was  employed  in  this  design  to  help  average  out  the  effect  of  the  many 
extraneous  variables  which  are  present  in  an  experiment  of  such  a  complex  and  diverse  activity 
as  PDM.  Other  factors  (such  as  which  system  a  subject  used  or  the  experience  of  the  subjects) 
were  purposefully  varied  in  the  plan  in  order  to  make  the  results  more  valid  for  a  large  variety  of 
situations  which  occur  in  the  general  population  of  all  the  ALCs.  By  using  factorial  designs,  the 
certain  factors  could  be  studied  in  one  experiment  as  well  as  the  interrelationship  between 
factors.  This  planning  effort  emphasized  three  important  developmental  phases  of  the  ITI-ALC 
project  in  Phase  U: 

1.  Experiment 

2.  Design 

3.  Analysis 

For  the  experiment,  much  effort  was  put  into  a  statement  of  objectives.  This  sotmds  like  an 
obvious  first  step;  however,  in  practice  it  often  takes  quite  a  while  to  get  general  agreement  as  to 
the  objective  of  any  long  term  effort.  Much  time  was  spent  reviewing  and  analyzing  all  pomts  of 
view  to  establish  just  what  the  Phase  n  experiment  will  be  intended  to  accomplish.  T^s 
carefully  constructed  statement  goes  a  long  way  to  define  the  Phase  n  effort  for  ITI-ALC.  ‘to 
compare  the  effectiveness  and  efficiency  of  mechanics  and  other  depot  PDM  personnel  when 
using  new  processes  and  technologies  described  in  the  ITI-ALC  System/Segment  Specification 
and  die  ITI-ALC  Business  Case. 

The  ITI-ALC  Demonstration/Evaluation  Plan,  Final,  dated  19  December  1995,  was  dedicated  to 
describe  the  “design”  of  the  experiment  for  the  ITI-ALC  Phase  n  project.  Many  times  an 
experiment  is  agreed  upon,  data  is  collected,  and  conclusions  are  drawn  with  little  or  no 
consideration  given  to  “how”  the  data  was  collected.  Questions  such  as  ‘‘How  mmy 
observations?”  --  “When?”  ~  “How  many  standard  deviations  are  needed  for  an  “outlier?”  - 
“What  is  the  size  of  the  population  and  what  size  is  needed  for  the  sample?”  -  “What  is  the 
order/sequence  of  experiments?”  The  influence  of  the  learning  effect  were  all  taken  into 
consideration  when  designing  the  experiments  for  Phase  n  of  ITI-ALC. 

The  final  step  taken  in  this  effort  was  analysis.  This  includes  the  approach  to  data  collection, 
type  of  data  to  be  collected,  data  reduction,  and  identification  of  the  statistics  to  be  used  m 
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making  decisions  about  various  aspects  of  the  experiment  In  this  case,  the  ^alysis  of  V^ance 
(ANOVA)  will  be  used  for  all  statistical  analysis.  Descriptive  and  inferential  analysis  will  also 
be  done  on  the  subjective  data  collected  with  forms  and  during  interviews. 

3.7.2  Demonstration/Evaluation  Plan  Validation 

The  only  formal  validation  for  the  resulting  documentation  of  this  effort  will  be  the  successful 
completion  of  the  demonstrations  cited  for  Phase  II  ITI-ALC.  The  document  and  the 
ideas/concepts  included  in  that  document  did  however  go  through  a  set  of  informal  and  formal 
reviews  as  part  of  the  standard  SRA  development  and  Quality  Assurance  (QA)  process. 


Ibis  process  typically  consists  of  two  or  three  events  with  additional  work  due  to  comments  from 
the  customer’s  formal  reviews  expected.  All  review  activities  are  documented  using  S^’s  QA 
Activity  Evaluation  Form.  The  ITI-ALC  Final  Demonstration/Evaluation  Plan  QA  reviews  was 
subject  to  follow  the  following  requirements. 

3.7.2.1  Deliverable  (Final)  Review 

This  review  checks  that  aU  action  items  from  previous  QA  activities  are  resolved  and  that  the  item 
is  ready  for  delivery.  This  review  should  take  place  (at  a  minimum)  two  days  prior  to  delivery.  The 
ITI-ALC  project  will  include  three  layers  of  quality  reviews. 

1 .  ITI-ALC  data  administrator(s) 

2.  QA  editor 

3.  Quality  Assurance  Official  (QAO)  -  “spot  checks” 

The  commanding  position  of  the  person  (Director  of  Dayton  Operations)  perfonmng  this  task 
will  ensure  autonomy  for  the  quality  review.  In  all  cases  specific  quality  criteria  will  be 
identified  (using  the  QA  Certification/Release  Form)  and  all  comments  will  be  in  written  form. 
The  criteria  for  acceptance  for  all  of  the  deliverables  on  the  ITI-ALC  project  will  be  taken 
the  section  of  the  SRA  QA  Manual  pertaining  to  acceptance  criteria.  The  form  will  be  kept  “on- 
file”  with  the  deliverable.  Generic  acceptance  criteria  can  be  found  included  in  Attachment  C  of 
the  ITI-ALC  Quality  Assurance  Plan  (QAP).  Specific  acceptance  criteria  will  be  developed  per 
deliverable  or  product  and  are  included  in  the  formal  documentation  folder  for  the  deliverable. 

3.7.2.2  Review  Based  on  Customer  Comments 

This  review  assesses  the  scope  of  the  customer's  comments  and  the  amount  of  rework  required. 
Significant  rework  will  require  that  an  additional  draft  review  be  held.  Materials  xised  for  this 
review  are  the  customer  comments  and  the  deliverable  itself. 

NOTE:  The  possible  recommendations  and/or  lessons  learned  on  this  subject,  it  did  not  seem  the 
customer  had  sufficient  resources  to  perform  a  significant  critique  of  this  document  or  effort. 


3.7.23  Peer  Review 

Peer  reviews  provide  early  visibility  into  the  status  and  quality  of  the  product  with  ample  time  to 
take  corrective  action  if  progress  and  quality  are  not  at  acceptable  levels.  This  review  checks  the 
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content  of  the  deliverable  to  assure  that  it  is  both  comprehensive  and  correct.  The  review  is 
performed  by  experts  in  the  subject  matter  as  well  as  “users”  of  the  document/product.  This  will 
ensure  that  the  documents  are  usable  as  well  as  comprehensive  and  compliant  with  requirements. 
Material  for  this  review  includes  the  draft  deliverable,  product  requirements/standards,  deliverable 
scorecard  that  allows  the  review  to  identify  strengths  and  weaknesses  within  particular  section(s), 
content  checklists  which  list  the  defined  success  criteria,  and  a  review  agenda.  For  CDRL  AGIO, 
the  peer  reviews  done  were  all  done  as  desk  top  reviews. 

3.7.2.4  Presentation 

For  many  documents  it  is  more  important  that  the  general  ITI-ALC  team  understand  the  concepts 
included  in  the  document  as  opposed  to  critiquing  the  document.  The  concepts  of  the  experiment 
documented  in  CDRL  AGIO  were  presented  to  the  group  as  a  whole  to  solicit  ideas  and  to  ensure 
everyone  understood  the  intent.  This  technique  also  helps  future  reviewers  understand  the  overall 
concept  before  reading  the  document. 

3.7.2.5  Audit 

An  audit  is  a  structured  examination/evaluation  by  the  QAO  of  a  defined  process  to  determine 
whether  or  not  the  process  is  being  implemented  in  accordance  with  established  guidelines  and 
whether  or  not  the  products  of  the  process  meet  established  acceptance  criteria.  The  QAO  will 
conduct  periodic  audits  of  the  various  processes  (e.g.,  configuration  management).  Prior  to  an 
audit,  or  process  review,  the  QAO  will  notify  all  parties  of  the  impending  audit,  describe  the  audit 
process  to  be  followed,  provide  a  schedule  for  all  audit  activities,  and  provide  a  copy  of  the 
evaluation  criteria.  The  QAO  will  conduct  the  audit  and  document  all  findings.  QA  will  then 
report  on  the  outcome  of  the  audit  If  no  problems  are  identified  or  only  minor  corrections  to  the 
process  are  required,  the  QAO  will  meet  with  the  parties  to  determine  appropriate  follow-up 
procedures  (e.g.,  verify  corrections  via  inspection  or  during  next  scheduled  audit).  If  the  results  of 
the  audit  indicate  that  major  corrections  to  the  process  are  required,  the  QAO  will  issue  a  Quality 
Action  Item  (QAI)  indicating  the  problem  and  providing  a  suspense  date  for  resolving  the  problem. 
Upon  completion  the  QAO  will  update  the  QA  database  with  audit  results  and,  if  necessary,  modify 
the  schedules  to  show  additional  audit  activity. 

3.7.3  Demonstration/Evaluation  Plan  Traceability 

The  traceability  requirement  for  the  demonstration  test  effort  is  a  trace  back  to  the  Business 
Process  Improvements  identified  as  part  of  the  overall  Phase  I  effort  of  ITI-ALC  and  documented 
in  the  ITI-ALC  Architecture  Report  and  the  Business  Case  for  ITI-ALC.  Table  3-3  lists  all  the 
BPIs  and  whether  they  relate  to  the  specific  demonstrations  and  tests.  The  traceability  will  help 
ensure  the  specific  benefits  of  a  BPI  can  be  measured  during  the  demonstration  and  that  the 
Business  Case  can  be  validated.  For  text  descriptions  of  this  BPIs,  reference  the  given  section  of 
the  ITI-ALC  Architecture  Report  and/or  the  ITI-ALC  Business  Case. 
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Table  3-3.  BPI  to  Demonstration/T est  Correlation 


# 

BPI  Recommendations 

AR* 

Ref. 

BC* 

Ref. 

Planning 

Test 

Debriefing 

Demo 

WP* 

Test 

TS* 

Demo 

1 

Process  and  Terminology  Coordination 

4.2.1 

C.1.1 

X 

X 

X 

X 

2 

Planning  Process  Enhancement 

4.2.2 

C.1.2 

X 

3 

Acquire  Parts 

4.2.3 

C.1.3 

X 

X 

X 

4 

Data  Sharing  Among  All  Levels  of  Maintenance 

MBM 

C.1.4 

X 

X 

X 

X 

5 

Production  Responsibility  Centers 

4.2.5 

C.1.5 

X 

X 

6 

Component  Parts  Acquisition  Policy  Changes 

4.2.6 

C.1.6 

X 

X 

X 

7 

Visibility  into  Part  Availability 

C.1.7 

X 

X 

X 

8 

Electronic  Signatures 

4.2.8 

C.1.8 

X 

X 

X 

9 

Performance  Metrics  Based  on  Actual  Data 

4.2.9 

C.1.9 

X 

10 

User  Technical  Information  Presentation  System 

4.2.10 

C.1.10 

X 

X 

X 

X 

11 

Preplanned  Over  &  Above/Unpredictables 

4.2.11 

C.1.11 

X 

X 

12 

Planning  Responsibility  Centers 

4.2.12 

C.1.12 

X 

1 

13 

Automated  and  Integrated  Technical  and 

Diagnostics  Information 

4.2.13 

C.1.13  i 

X 

14 

Multi-skilled  Mechanics 

4.2.14 

C.1.14 

X  1 

15 

Three  Shifts 

N/A 

Cl. 15 

^  .  _.a.  J 

.4 /vvn  /' 

3.7.4  Demonstration/Evaluation  Plan  Possible  Recommendations  and  Lessons  Learned 

The  following  observations  were  made  as  suggestions  to  improving  the  results  and  efficiency  of 
future  efforts  of  this  nature. 

•  The  title  of  the  resulting  document  for  this  effort  should  have  been;  Experiment  Design 
Document  for  the  Demonstration  of  the  Efficacy  of  the  ITI-ALC  System  and  BPIs. 

It  was  obvious  from  ITI-ALC’s  Statement  of  Work  (SOW)  that  AL/HRGO  wanted  a  plan  for 
an  experiment  demonstrates  the  improved  effectiveness  and  efficiency  of  mechamcs  and 
other  depot  PDM  personnel  when  using  new  processes  and  technologies  described  in  the 
System/Segment  Specification  and  the  ITI-ALC  Business  Case.  This  intent  was  not  obvious 
from  the  title  of  the  document  per  the  CDRL  list.  A  “demonstration  denotes  a  certain 
informal  approach  to  the  effort  and  a  “test”  would  most  likely  be  confused  with  a  System 
Test”  which  has  a  different  intent  (the  intent  of  all  formal  system/software  testing  is  to  find 
errors  in  the  system/software  [Rubey86])  then  what  will  be  done  for  Phase  II  of  ITI-ALC.  In 
the  future,  AL/HRGO  should  name  similar  efforts  as  “Experiments  . 


•  Adjust  the  development  schedule  for  the  Demostration  Plan. 

Before  time  and  resources  are  spent  on  designing  an  experiment,  customer  resources  should 
be  allocated  to  the  review,  analysis,  understanding  and  critiquing  of  the  results  of  that  effort. 
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If  another  such  program  is  conducted  in  the  future,  AL/HRGO  should  consider  removing  this 
tasks  from  the  Phase  I  SOW  due  to  it’s  premature  nature.  This  task  is  better  suited  to  a  time 
closer  to  the  actual  implementation  of  the  plan. 

•  Cost  consideration  for  the  experiment  should  be  included. 

The  test  or  demonstrations  identified  in  the  deliverable  for  this  effort  did  not  presume  to 
evaluate  the  cost  of  performing  the  tasks  identified  in  the  experiment  or  the  cost  of  building  a 
demonstration  system  to  conduct  those  tasks.  However,  the  design  was  developed  so  that  one 
half  of  the  experiment  could  be  conducted  with  little  of  no  interaction  with  the  second  half, 
therefore  allowing  for  a  truncated  test/demonstration.  Another  way  to  do  this  would  be  to  get 
potential  users  involved  much  earlier  (e.g.,  at  the  second  release  of  the  SSS)  in  the  process  to 
define  which  functional  requirements  they  would  like  to  see  in  a  test/demonstration.  This 
information  would  then  have  to  be  made  available  to  the  contractor  performing  the 
experiment  design  effort. 

•  More  emphases  on  functional  demonstration  verses  system  interface. 

To  ensure  a  cost  effective  demonstration,  more  emphases  should  be  put  on  prototyping  the 
functional  capabilities  of  the  ITI-ALC  system  verses  making  something  that  will  evolve  into 
a  production  system.  One  aspect  of  “production  level”  software  is  that  too  much 
time/resources  are  spent  in  building  “real”  system-to-system  interfaces  with  legacy  and 
emerging  system.  Most  of  the  data  obtained  from  the  supporting  system  included  in  the  ITI- 
ALC  design  can  be  simulated/derived,  allowing  for  more  resources  devoted  to  building  a 
system  that  helps  the  user  imderstand  the  ITI-ALC  BPI  concepts  and  that  is  flexible  enough 
to  try  many  different  ideas.  The  industries  trend  toward  “evolutionary”  prototypes  is  an 
unrealized  potential  because  the  nature  of  prototypes  and  the  nature  of  “production”  software 
are  mostly  mutually  exclusive.  This  lesson  highlights  and  utilizes  the  flexibility  and 
modularity  of  the  ITI-ALC  design  per  the  SSDD  and  the  SM. 
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4.  CONCLUSIONS  AND  RECOMMENDATIONS 


The  rri-ALC  program  was  performed  to  develop  the  requirements  for  a  more  efficient, 
streamlined  organic  PDM  process  by  using  a  two  step  BPR  approach.  The  first  set  w^  to 
document  and  analyze  the  current  process.  Through  this  analysis,  the  process  was  stre^lined, 
the  informational  requirements  to  support  the  process  was  refined,  the  organizational 
responsibilities  for  the  process  segments  was  revised,  and  the  regulation  modifications  needed  to 
support  the  streamlined  process  were  identified.  The  second  set  was  to  develop  the  requirements 
for  computerized  m-ALC  system  that  will  effectively  support  the  implementation  of  the 
streamlined  PDM  process.  This  section  stunmarizes,  as  required  by  the  DID  referenced  by  the 
SOW,  the  discussion  and  recommendations  presented  in  this  report. 

The  ITI-ALC  BPR  analysis  program  was  effectively  and  efficiently  performed  through  Ae  use  of 
SRA's  user-focused,  structured  BPR  methodology  implemented  using  a  proven  set  of  integrated 
tools  and  techniques.  This  methodology  ensured  a  high  probability  of  success  because  it 
established  an  integrated  team  consisting  of  SRA  team  members,  functional  experts,  HRGO 
program  personnel,  and  users  firom  each  of  the  five  ALCs.  Specifically,  this  methodology 
focused  on  the  PDM  process  firom  the  user's  perspective,  thereby  ensuring  that  user  problems 
were  identified  and  that  the  resulting  improvement  concepts  addressed  the  user  problems,  were 
practical,  were  accepted  by  the  users,  and  were  incorporated  into  the  requirements  for  the  ITI- 
ALC  system  and  traceable  back  to  the  user  requirements 

The  users  that  were  actively  involved  in  the  performance  of  the  ITI-ALC  program  represented  all 
aspects  the  depot  maintenance  process  contained  within  the  scope  defined  for  the  program, 
represented  all  five  of  the  ALCs,  and  represented  the  maintenance  process  used  to  maintain  a 
variety  of  systems  and  aircraft  type.  From  each  ALC,  the  users  included  planners,  sch^ulers, 
aircraft  managers,  and  mechanics,  as  well  as  a  variety  of  individuals  responsible  for  activities  that 
interface  with  these  primary  users,  such  as  supply.  The  items  used  as  the  select  set  for  the 
analysis  included  a  combination  of  light  and  heavy  aircraft  as  well  as  engines.  Specifically, 
included  were  the  F-16,  F-15,  F-22,  C-135,  and  C-5  aircraft;  and  the  FI  10  and  FlOO  engines. 
Within  these  items,  those  maintenance  processes  on  which  the  analysis  was  specifically  directed 
were  for  the  EF-111  Tactical  Jamming  System,  LANTERN,  Advanced  Composites,  Landing 
Gear,  Fuel  Controls,  Constant  Speed  Drives,  and  Generators. 

Figure  4-1  provides  an  overview  of  the  methodology  used  to  perform  the  PDM  process  analysis. 
This  methodology  was  developed  over  a  period  of  fifteen  years  by  SRA  team  members,  is  well 
documented  in  a  number  of  published  papers  and  case  studies,  and  has  been  successfully  applied 
on  a  number  of  BPR  efforts.  While  we  have  used  this  methodology  successfully  to  iniprove 
processes,  we  view  the  methodology  and  the  associated  techniques  and  tools  as  an  evolutionary 
process.  Therefore,  as  part  of  each  application,  we  analyze  and  refine  the  methodology, 
techniques,  and  tools  based  on  the  possible  recommendations  and/or  lessons  learned  from  each 
application.  Using  this  approach  to  process  improvement  analysis,  we  are  continually  increasing 
the  probability  of  success  for  each  BPR  analysis  effort. 
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Figure  4-1.  The  Enterprise  Representation  and  Analysis  Process 

While  reading  this  discussion  of  the  methodology,  it  is  important  to  understand  that  the  models 
developed  through  the  application  of  this  methodology  are  not  final  products,  but  rather  are 
critical  analysis  tools  that  provide  the  foundation  needed  for  understanding,  documenting  and 
analyzing  the  depot  maintenance  process.  Through  the  analysis  of  the  models  are  generated  the 
accurate  and  complete  SSS  and  SSDD  needed  to  take  the  next  step  of  design,  developing,  and 
implementing  the  improved  process  and  the  ITI-ALC  system. 

The  BPR  improvement  methodology  begins  with  the  definition  of  the  problem  to  be  addressed. 
This  definition  defines  the  enterprise  to  be  examined  and  the  improvement  goals  to  be  achieved. 
The  problem  definition  is  analyzed  to  establish  the  specific  domain  of  the  enterprise  to  be 
analyzed,  the  purpose  and  viewpoint  around  which  the  enterprise  is  to  be  analyzed,  the 
objectives  of  the  Business  Process  Reengineering  effort,  and  the  evaluation  criteria  to  be  used  in 
judging  the  effectiveness  of  the  BPIs. 

The  domain,  purpose,  and  viewpoint  of  the  enterprise  then  guide  the  development  of  a  set  AS- 
IS"  models  that  provide  a  static  description  of  the  process  firom  functional,  data,  and  cost 
perspectives.  The  initial  development  of  these  models  help  to  establish  and  scope  the  type  of 
data  needed  to  describe  the  enterprise,  to  identify  the  sources  for  the  data,  to  collect  the  data,  and 
to  organize  and  access  the  data.  These  sources  may  include  functional  area  experts,  users, 
enterprise  documentation,  etc.  These  collected  data  are  used  to  refine  and  expand  the  AS-IS 
models  into  a  complete  and  accurate  description  of  the  current  process. 


69 


To  complete  the  BPR  analysis  also  requires  the  analysis  of  the  process  from  a  performance 
perspective.  This  perspective  is  developed  by  using  the  "AS-IS"  functional  model  as  the  tois 
for  identifying  and  collecting  process  performance  information,  developing  a  process  flow 
modeling  onto  which  is  overlaid  the  performance  information  to  produce  a  simulation  which  can 
then  be  exercised  to  assist  in  the  verification  and  validation  of  the  "AS-IS"  model  set,  and  to 
analyze  the  performance  of  the  current  process.  At  a  minimum,  these  dynamic  charactenstics 
include  functional  timing  specifications  and  the  process  flow  through  the  enterprise. 

Through  the  analysis  of  the  "AS-IS"  functional,  data,  cost,  and  performance  models  improvement 
concepts  are  identified  and  documented  using  a  corresponding  set  of  "TO-BE"  models.  When 
completed,  the  "TO-BE"  models  provide  the  foundation  for  developing  the  description  for  the 
ITI-ALC  system  that  will  support  the  implementation  of  the  improved  process  as  documented  m 
the  "TO-BE"  model  set.  The  description  of  the  ITI-ALC  system  begins  by  extracting  the 
requirements  from  the  "TO-BE"  models  and  documented  them  in  the  SSS.  A 
operational  design  for  the  ITI-ALC  system  is  developed  by  developing  a  system  model  which 
describes  how  the  various  proposed  functional  components  the  m-ALC  system  will  mteract  to 
share  data  throughout  the  performance  of  the  depot  maintenance  process.  This  proposed 
operational  concept  is  then  documented  in  the  SSDD. 

Resulting  from  the  methodology,  therefore  are  a  set  of  "AS-IS"  and  "TO-BE"  mcxiels,  and 
specifications  for  the  improved  process  and  for  the  ITI-ALC  system  documented  via  Ae  system 
model,  the  SSS,  and  the  SSDD.  These  specifications  provide  the  information  foundation  that 
will  be  required  durmg  the  design  and  development  phase. 

An  integrated  set  of  techniques,  supported  by  automated  tools,  was  applied  to  effectively 
implement  the  methodology.  The  specific  products,  and  the  techniques  used  to  develop  the 
products,  are  summarized  as  follows: 

Functional  Models  fFMs)  —  IDEF© 

“AS-IS”  -  Represents  the  current  depot  maintenance  activities. 

“TO-BE”  -  Represents  the  recommended  or  future  depot  maintenance  activities. 

Data  Models  (DMs)  —  IDEFu  *  j  * 

“AS-IS”  -  Represents  the  logical  information  structure  that  supports  the  current  depot 

maintenance  process. 

“TO-BE”  -  Represents  the  recommended  logical  information  structure  to  support  the 
future  dqpot  maintenance  process. 

Process  Models  tPM)  —  IDEF3 

“AS-IS”  -  Represents  the  dynamic  aspects  of  the  current  depot  mamtenance  activities 
as  defined  by  the  “AS-IS”  FM. 

“TO-BE”  -  Represents  the  dynamic  aspects  of  the  recommended  or  future  depot 
maintenance  activities  as  defined  by  the  ‘  TO-BE  FM. 
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Simulation  Models  —  WITNESS 

Once  developed,  the  IDEF3  models  were  exercised  to  represent  the  operational 
characteristics  of  the  PDM  process.  Specifically,  the  simulation  models  provided  the 
following: 

"AS-IS"  -  Represents  the  performance  aspects  of  the  current  PDM  process  as  defined 
in  the  "AS-IS"  fimctional  model. 

"TO-BE"  -  Represents  the  performance  aspects  of  the  recommended  or  future  PDM 
process  as  defined  by  the  “TO-BE”  FM. 

System  Model  (SM)  -  Data  Flow  Diagrams 

Defines  an  operational  concept  for  the  ITI-ALC  system  and  provides  transition  from  the 
requirements  definition  phase  to  the  design  and  development  phases. 

Business  Case  (BCI  —  CDRL  A002 

-  Documents  the  cost/benefit  analysis  for  the  “TO-BE”  implementation. 

Svstem/Segment  Specification  (SSS)  —  CDRL  A008 

-  Documents  the  requirements  for  the  retool  needed  in  the  “TO-BE” 

implementation. 

Svstem/Sefnuent  Design  Document  (SSPD)  —  CDRL  A014 

-  High-level  design  for  the  “TO-BE”  implementation. 

IMIS/ITI-AT.C  System  Comparison  -  CDRL  AOll 

-  Functional  comparison  between  the  IMIS  and  ITI-ALC  system,  and  a 
demonstration  plan  for  the  next  phase  of  ITI-ALC. 

Demonstration  Plan  —  CDRL  A012 

-  Planning  document  for  a  Phase  11  demonstration  of  the  ITI-ALC  system. 

As  represented  in  Figure  4-2,  the  development  of  the  various  "AS-IS"  and  "TO-BE"  models,  the 
system  model,  the  business  case,  the  SSS,  the  SSDD  were  developed  in  an  iterative  manner  with 
the  users  providing  key  participation  throughout  the  program.  This  ensured  consistency  and 
completeness  among  the  various  models  and  documents,  and  ensured  the  tractability  of 
information  all  the  way  firom  the  users,  through  the  various  models,  and  into  the  SSS  and  SSDD. 

The  problem  definition  provided  the  basis  for  the  development  of  the  "AS-IS"  models  which 
were  validated  with  the  users.  Through  the  analysis  of  these  models,  process  improvement 
opportunities  were  identified  and  used,  along  with  improvement  ideas  provided  by  the  users^o 
refine  the  "AS-IS"  models  into  the  "TO-BE"  models  representing  the  streamlined  process.  The 
"TO-BE"  models  and  improvement  concepts  were  presented  to  the  users  to  gain  their 
xuiderstanding,  review,  comments,  and  acceptance. 
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PROBLEM 

DEFINITION 


Figure  4-2.  The  BPR  Analysis  Methodology  is  User  Focused 

When  completed,  the  "TO-BE"  models  provided  the  basis  for  documenting  the  requirements  as 
represented  in  the  models  and  to  begin  an  initial  implementation  design  of  the  improved  process 
and  ITI-ALC  system.  The  implementation  of  the  improvement  concepts  had  to  satisfy 
implementation  requirements  consisting  of  changes  in  processes,  policies  and  procedures, 
organizations,  and  environment  and  interface  changes.  As  with  the  models  and  improvement 
opportunities,  the  documented  requirements  and  initial  design  were  validated  with  the 
government  personnel  and  users. 

The  users  involvement  ensured  the  overall  BPR  analysis  effort  maintained  its  user  focus  and 
ensured  user  acceptance  and  therefore  success  of  the  ITI-ALC  program.  The  roles  played  by  the 
users  are  summarized  as  follows; 

1.  Provided  information  about  the  maintenance  process  from  each  of  their  perspectives  and 
was  used  as  the  starting  point  for  understanding  and  documenting  the  current  process. 

2.  Provide  their  perspectives  about  the  problems  that  exists  within  the  current  process  and 
for  improvement  ideas  that  could  help  the  operational  effectiveness  of  the  current  process. 

3.  Reviewed  the  "AS-IS"  models  to  ensure  that  a  complete  and  accurate  foundation  had 
been  developed  on  which  to  build  the  analysis  effort. 
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4.  Reviewed,  refined,  and  commented  on  the  improvement  concepts  as  they  were  being 
formalized  to  ensure  they  would  help  the  users  and  would  provide  practical  benefits. 

5.  Reviewed  and  commented  of  the  "TO-BE"  models  to  ensure  that  required  process 
fimctionality  had  not  been  lost  or  that  unnecessary  fimctionality  had  been  added. 

6.  Reviewed  the  SM  to  verify  that  tine  operational  concept  presented  in  the  SM  satisfied  the 
operational  goals  of  the  maintenance  personnel. 

7.  Reviewed  the  SSS  to  ensure  that  the  requirements  were  complete,  accurate,  and  stated  in 
a  manner  that  was  understandable  to  file  maintenance  community. 

While  the  ITI-ALC  program  requirements  were  satisfied  with  the  development  of  the  SSS  and 
SSDD,  the  next  phase  in  the  overall  ITI-ALC  program  is  the  completion  of  the  design,  followed 
by  the  development  and  implementation  of  the  improved  process  and  the  ITI-ALC  system.  The 
usgi'-focused  information  developed  thus  far  will  provide  a  solid  foundation  on  which  to  perform 
the  next  phase.  Reasons  for  the  solid  foimdation  are  the  following: 

1 .  The  users  have  been  identified. 

2.  The  users  are  fully  award  of  and  have  accepted  the  concepts  that  are  to  be  implemented. 

3 .  Specific  user  problems  are  being  addressed. 

4.  A  complete  set  of  requirements  exist  to  guide  the  design  and  development. 

Using  this  approach  reduces  the  possibility  that  the  system  designer  will  under  or  over  specify  the 
support  system.  Under-specifying  the  support  system  will  reduce  user  acceptance  of  the  system; 
whereas  over-specifying  will  likely  increase  the  cost  of  the  implementation,  and  reduce 
utilization  of  the  support  systems.  Through  the  accurate  specifications  for  the  support  system,  a 
good  correlation  between  the  processing  requirements  and  the  technology  capabilities  can  be 
derived,  resulting  in  an  efficient,  more  cost  effective  enterprise. 

In  conclusion,  the  method  used  to  fulfill  the  ITI-ALC  Phase  I  requirements,  although  not  perfect 
(as  indicated  be  the  numerous  recommendations  and/or  lessons  learned  throughout  this  report),  is 
a  very  efficient  and  effective  process  to  perform  BPR.  It  is  recommended  that  the  items 
identified  as  lessons  be  phased  into  the  baseline  process  over  several  programs  based  on  medics 
focused  benefit/cost  analysis  of  each  of  the  improvements.  Appendix  B  lists  the  possible 
recommendations  and/or  lessons  learned  organized  by  project  entity  (e.g.,  FM,  DM,  SSS,  SSDD, 
and  so  forth).  The  newest  “golden  bullets”  although  exciting,  are  always  accompanied  by  high 
risk,  rework  and  cost  ~  revolution  is  seldom  without  pain.  Given  this,  steady,  well  planned 
evolution  based  on  experience  and  a  balance  between  new  and  old  ideas  is  preferred  unless 
customer  expectations  are  equal  to  the  reality  of  the  revolutionary  process. 


73 


5,  REFERENCES 


This  section  lists  reference  citations  that  are  used  within  this  document.  The  list  of  references 

below  are  alphabetically  arranged. 

•  CIM  Technical  Reference  Model,  Version  1.2,  May  15, 1992. 

•  DoD  8320.1-M-l,  DoD  Data  Element  Standardization  Procedures,  January  1993,  authorized 
by  DoD  Directive  8320.1,  September  26, 1991. 

•  DoD  TAFIM,  Volume  1,  Overview,  Version  2.0,  Jime  30,  1994. 

•  DoD  TAFIM,  Volume  2,  Technical  Reference  Model  and  Standards  Profile  Summary, 
Version  2.0,  Jime  30, 1994. 

•  DoD  TAFIM,  Volume  3,  Architecture  Concepts  and  Design  Guidance,  Version  2.0,  June  30, 
1994. 

•  DoD  TAFIM,  Volume  4,  DoD  Standards-Based  Architecture  Planning  Guide,  Version  2.0, 
June  30, 1994. 

•  DoD  TAFIM,  Volume  6,  DoD  Goal  Security  Architecture,  Version  2.0,  June  30, 1 994. 

•  DoD  TAFIM,  Volume  8,  DoD  HCL  Style  Guide,  Version  2.0,  June  30, 1 994. 

•  FIPS  Publication  161,  Electronic  Data  Interchange  (EDI),  April  19, 1993. 

•  FIPS  Publication  179,  NIST  Government  Network  Management  Profile  (GNMP)  Draft, 
December,  1992. 

•  FIPS  Publication  183,  Integration  Definition  for  Function  Modeling  (IDEFO),  December  21, 
1993. 

•  FIPS  Publication  184,  Integration  Definition  for  Information  Modeling  (IDEFIX),  December 
21, 1993. 

•  GAO-GGD-94-154,  Proposed  Policy  to  Accept  Credit  and  Debit  Cards  Make  Sense 
Conceptually,  Arthur  D.  Little,  1994. 

•  GAO/AFMD-92-12,  Financial  Audit:  Aggressive  Actions  Needed  for  Air  Force  to  meet 
Objectives  of  the  CFO  Act,  1992. 

•  GAO/NSIAD-91-176,  Defense  Inventory:  Shortcomings  in  Requirements  Determination 
Processes,  1991. 


74 


•  nCE,  IDEF3  Process  Description  Capture  Method  Report,  May  1992, 

•  IMIS,  Final  Scientific  and  Technical  Report  of  the  IMIS  Architecture,  General  Dynamic 
Electronics  Division/SofTech  Inc.,  30  May  1990. 

•  ITI-ALC,  Final  Architecture  Report,  Systems  Research  and  Applications  Corporation, 
Contract  No.  F41624-94-C-5021,  CDRLNo.  A007, 29  June  1995. 

•  ITI-ALC,  Business  Case  -  Final  Submittal,  Systems  Research  and  Applications  Corporation, 
Contract  No.  F41624-94-C-5021,  CDRL  No.  A002, 29  March  1996. 

•  ITI-ALC,  Depot  Requirements  Compared  to  Prototype  Capability,  Systems  Research  and 
Applications  Corporation,  Contract  No.  F41624-94-C-5021,  CDRLNo.  AOll,  19  December 
1995. 

•  ITI-ALC,  Demonstration/Evaluation  Plan,  Systems  Research  and  Applications  Corporation, 
Contract  No.  F41624-94-C-5021,  CDRLNo.  AGIO,  19  December  1995. 

•  ITI-ALC,  System/Segment  Design  Document  -  Final,  Systems  Research  and  Applications 
Corporation,  Contract  No.  F41624-94-C-5021,  CDRLNo.  A014, 15  February  1996. 

•  ITI-ALC,  System/Segment  Specification  -  Final,  Systems  Research  and  Applications 
Corporation,  Contract  No.  F41624-94-C-5021,  CDRLNo.  A008,  31  October  1995. 

•  JLSC’s  Improved  Functional  Baseline  (IFB)  IDEFO  Model  and  Glossary,  October  1993, 
Revised  July  1994. 

•  JLSC’s  Improved  Functional  Baseline  (IFB)  IDEFIx  Model  and  Glossary  (key-based) 
Technical  Report,  February  1994. 

•  MIL-D-87269,  Military  Specification,  Data  Base,  Revisable:  Interactive  Electronic 
Technical  Manuals,  for  the  Support  Of,  20  November  1992. 

•  MR-313-A/USN,  BJIFID,  An  Approach  to  Understanding  the  Value  of  Parts,  April  1991. 

•  Strategy  and  Structure,  A.  D.  Chandler,  Cambridge  Mass,  MIT  Press,  1992. 

•  The  DoD  Enterprise  Model,  Volume  I:  Strategic  Activity  and  Data  Models,  January  1994. 

•  The  DoD  Enterprise  Model,  Volume  II:  Using  the  DoD  Enterprise  Model  -  A  Strategic  View 
of  Change  in  DoD  -  A  White  Paper,  January  1994. 


75 


APPENDIX  A.  MODEL  DOCUMENTATION 

The  significant  information  contained  in  the  models  and  the  resources  invested  in  the  various 
models  required  they  be  effectively  documented  for  use  on  Ae  ITI-^C  program  zs  well  as 
references  on  future  programs.  These  models,  and  their  associated  information  were 
documented  in  the  Appendices  of  the  ITI-ALC  Architecture  Report  as  follows: 


APPENDIX  CONTEXT 


A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

O 

P 

Q 


“AS-IS”  Functional  Model 
“AS-IS”  Data  Model 
“AS-IS”  Data  Glossary 

(For  the  Functional  and  Data  “AS-IS”  models) 

“TO-BE”  Functional  Model 

“TO-BE”  Data  Model 

“TO-BE”  Data  Glossary 

(For  the  Functional  and  Data  “TO-BE”  models) 

System  Model 
System  Model  Glossary 
“AS-IS”  Process  Model 
“TO-BE”  Process  Model 

“AS-IS”  Functional  Model  to  Interview  Traceability  Matrix 

Date  Model  to  “AS-IS”  Functional  Model  Traceability  Matrix 
“TO-BE”  to  “AS-IS”  Functional  Model  Traceability  Matrix 
“TO-BE”  Date  Model  to  Functional  Model  Traceability  Matrix 
System  Model  to  ^*TO-BE”  Functional  Model  Traceability  M^atrix 
“TO-BE”  Functional  Model  to  the  IMIS  “TO-BE”  Control  Architecture 
Traceability  Matrix 

Interview  Kit  used  during  the  Date  Collection  Trips 


The  development  of  each  of  these  appendices  was  accomplished  using  automated  techniques. 
The  automated  documentation  development  approaches  ensured  the  completeness  and  accmacy 
of  each  appendix,  reduced  the  level  of  resources  needed  to  produce  the  documen^  ensiued  the 
consistent  and  high  visual  quality  of  the  document,  and  allowed  for  the  flexibility  of 
incorporation  of  any  model  refinement  up  to  the  last  minute. 


A.1  FUNCTIONAL  MODEL  (FM)  DOCUMENTATION 

The  documentation  procedure  for  the  "AS-IS"  and  "TO-BE"  functional  models,  developed  using 
Design/IDEF,  is  represented  in  Figure  A-1.  The  node  list  was  developed  by  processing  the 
activity  report  produced  by  Design/IDEF. 
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The  page-pair  format  of  the  functional  model  was  developed  by  aggregating  the  model  diagrams 
obtained  from  Design/IDEF  and  the  associated  narrative  text  obtained  from  a  word  processor. 
To  accomplish  this,  the  structure  of  the  model  was  used  to  set  up  the  page-pair  format.  Then  the 
diagram  graphics  from  Design/IDEF  were  inserted  into  the  appropriate  locations,  and  the 
narrative  text  was  inserted  to  complete  the  page-pair  documentation,  as  contained  in  Appendices 
AandD. 

To  complete  the  documentation  of  the  functional  models  required  the  development  of  then- 
respective  glossaries.  A  list  of  all  the  arrow  names,  their  diagram  references,  and  a  listing  of  all 
arrow  definitions  were  extracted  from  Design/IDEF.  These  three  sets  of  information  were  then 
merged  to  form  the  functional  model  portion  of  the  glossaries  documented  in  Appendices  C  and 

F. 


Figure  A-1.  The  FM  Documentation  Procedure 

A.2  DATA  MODEL  (DM)  DOCUMENTATION 

The  documentation  of  the  data  models  presented  a  significant  challenge.  A  complete  data  model 
for  a  process  such  as  the  depot  maintenance  is  larger  than  can  effectively  be  displayed.  Also,  the 
entity  diagram  approach  used  on  other  programs  has  not  proved  to  be  an  effective 
documentation  technique  because  it  is  too  restrictive  of  the  information  available  to  the 
reader/analyst  at  any  point  in  time. 

To  improve  the  effectiveness  of  the  data  model  documentation,  subject  areas  or  views  into  the 
complete  data  model  were  selected  and  presented  in  a  page-pair  format  to  augment  a  foldout  of 
the  entire  model.  These  subject  areas  limited  the  amount  of  information  so  that  it  could  be 
effectively  displayed  on  an  8.5”  x  11”  page  but  included  a  grouping  of  information  that 
described  a  particular  aspect  of  the  process. 
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To  facilitate  the  selection  and  documentation  of  these  views  required  the  use  of  computerized 
capabilities.  While  the  final  documentation  of  the  "AS-IS"  and  "TO-BE"  data  models  have  the 
same  appearance,  the  tools  used  to  develop  each  of  the  models  were  different.  As  a  resdt,  the 
procedures  for  developing  the  final  documentation  for  each  of  the  models  were  slightly  different 
and  are  described  in  the  following  paragraphs. 

A.2.1  Documentation  of  the  ”  AS-IS"  Data  Model 

The  "AS-IS"  data  model  was  developed  using  Design/IDEF  and  Figure  A-2  represents  the 
documentation  process  that  was  applied.  The  views  into  the  data  model  were  identified  and  then, 
extracted  from  the  completed  model  through  the  processing  of  the  SML  report.  The  extracted 
subject  area  SML  was  then  processed  through  Design/IDEF  and  inserted  mto  the  appropnate 
location  within  the  page-pair  document.  The  narrative  description  for  each  view  w^  then 
developed  and  added  to  complete  the  page-pair  documentation  of  the  "AS-IS"  data  model.  This 
page-pair  document  was  placed  in  Appendix  B  of  the  Architecture  Report. 


To  facilitate  locating  entities  within  the  various  subject  diagrams  and  to  help  ensure  tha^l 
entities  had  been  included  in  at  least  one  view,  a  cross  reference  report  was  developed.  This 
report  was  developed  by  processing  the  IDEFix  report  generated  by  Design/IDEF  and  was 
placed  in  Appendix  B. 
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The  glossary  was  developed  to  identify  and  define  each  entity  and  attribute  within  the  "AS-IS 
data  model.  The  glossary  was  developed  by  processing  the  glossary  report  produced  by 
Design/IDEF.  This  glossary  was  then  merged  with  the  glossary  for  the  Ab-lb  lunctionai 
model  to  complete  the  development  of  Appendix  C  within  the  Architecture  Report. 


A.2.2  Documentation  of  the  ”TO-BE’’  Data  Model 


While  the  capabilities  of  Design/IDEF  to  support  the  "AS-IS"  data  modelmg  were  adequate,  an 
analysis  of  the  ERWin  tool  demonstrated  that  it  had  key  benefits  over  the  data  modelmg  aspects 
of  Design/IDEF.  Specifically,  ERWin  provided  improved  of  attributes  among  the  entities  to 
better  s^port  the  implementation  of  the  IDEF,x  notation.  ERWin  also  facilitated  tae 
development  of  the  glossary  and  was  effective  in  pullmg  out  the  subjects  areas  identified  for 
model  documentation.  Therefore,  ERWin  was  selected  for  use  in  developing  the  TO-BE  data 

model. 


Because  of  the  limited  graphics  control  provide  by  ERWin,  the  completed  model  ^feti^ 
from  ERWin  into  Design/IDEF  where  the  model  layout  was  adjusted  to  provide  the  quality 

needed  for  the  Architecture  Report. 


A.3  PROCESS  MODEL  (PM)  DOCUMENTATION 

The  documentation  of  the  process  model  is  made  of  two  components  which  are  contamed  m 
Appendices  I  and  J  for  the  "AS-IS"  and  "TO-BE"  models,  respectively.  These  components  ^e 
the  diagrams  and  the  activity  performance  information.  The  flow  of  information  into  the 
Architecture  Report  and  the  structure  of  these  appendices  are  represented  in  Figure  A-3. 


Figure  A-3.  The  PM  Documentation  Process 
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A.4  SYSTEM  MODEL  (SM)  DOCUMENTATION 

The  system  model,  which  addresses  the  design  of  the  ITI-ALC  depot  maintenance  system,  was 
developed  using  System  Architecture.  The  model  was  developed  using  data  flow  diagrams 
which  are  hierarchical  in  nature  much  like  the  IDEFo  model.  Therefore  the  p^e-pair  format 
consisting  of  the  diagrams  and  associated  text  was  chosen  for  the  SM.  This  page-p^  format  is 
then  supplemented  with  a  node  list  and  glossary.  The  procedure  for  developing  the  SM 
documentation  is  represented  in  Figure  A-4. 


Figure  A-4.  The  SM  Documentation  Process 

A  customized  set  of  report  generators  were  developed  to  select  and  transfer  information  from 
System  Architecture  into  an  ASCH  file  for  documenting  the  SM.  One  of  these  reports  was  used 
to  establish  the  page-pair  format  for  the  report.  The  diagrams  from  System  i^chitecture  were 
extracted  and  inserted  into  the  page-pair  structure.  The  narrative  for  each  diagram  was  then 
inserted  to  complete  the  page-pair  format  of  the  SM  as  contained  in  Appendix  G  of  the  System 
Architecture. 

Another  customized  report  extracted  the  process  list  from  System  Architecture  and  processed  to 
produce  the  node  list  for  the  system  model.  This  node  list  is  located  in  front  of  the  model  in 

Appendix  G. 

The  final  step  in  documenting  the  system  model  was  the  generation  of  the  glossary  for  the  data 
flows.  The  names  of  the  data  flows  were  extracted  from  System  Architecture  by  a  customized 
report  and  definitions  added  to  the  flow  names.  The  definitions  and  the  ^ta  flow  names  were 
then  formatted  into  the  glossary  as  documented  in  Appendix  H  of  the  Architecture  Report. 
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APPENDIX  B.  ACRONYM/ABBREVIATION  LIST 

This  appendix  lists  the  acronyms  and/or  abbreviations  included  in  this  document. 


Acronym 

Abbreviation 

Deflnition 

A/C 

Aircraft 

AFLC 

Air  Force  Logistics  Command 

AFMC 

Air  Force  Materiel  Command 

AFTO 

Air  Force  Technical  Order 

AHP 

Analytic  Hierarchy  Process 

AIS 

Automated  Information  System 

ALC 

Air  Logistics  Center 

AL/HRGO 

Armstrong  Laboratory/Logistics  Research  Division, 
Operational  Logistics  Branch 

ANOVA 

Analysis  of  Variance 

APDS 

Automatic  Parts  Distribution  System 

ASRS 

Automated  Storage  and  Retrieval  System 

BOM 

Bill  of  Materials 

BPI 

Business  Process  Improvement 

BPR 

Business  Process  Reengineering 

CALS 

Continuous  Acquisition  and  Life-Cycle  Support 

CAMS 

Core  Automated  Maintenance  System 

CDR 

Critical  Design  Review 

CDRL 

Contract  Data  Requirements  List 

CEMS 

Comprehensive  Engineering  Management  System 

CIM 

Corporate  Information  Management 

CMM 

Capability  Maturity  Model 

COTS 

Commercial-off-the-Shelf 

CSC 

Computer  Software  Component 

D-level 

Depot  level 

DFD 

Data  Flow  Diagram 

DM 

Data  Model 

DM-HMMS 

Depot  Maintenance-Hazardous  Material  Management  System 
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Acronym 

Abbreviation 

DMMIS 

Definition 

Depot  Maintenance  Management  Information  System 

DMSS 

Depot  Maintenance  Standard  System 

DoD 

Department  of  Defense 

ESD 

End-Item  Status  Display 

FCA 

Functional  Configuration  Audit 

FEMS 

Facility  Equipment  Management  System 

FIPS 

Federal  Information  Processing  Standards 

FM 

Functional  Model 

FSS 

Financial  Standard  System 

HAZMAT 

Hazardous  Material 

HW 

Hardware 

HMSS 

Hazardous  Material  Standard  System 

HWCI 

Hardware  Configuration  Item 

ICD 

Interface  Control  Document 

ICN 

ITI-ALC  Communication  Network 

IDEF 

Integrated  DEFinition 

BETM 

Interactive  Electronic  Technical  Manuals 

I/F 

Interface 

I-level 

Intermediate  level 

IMDS 

Integrated  Maintenance  Data  System 

IMIS 

Integrated  Maintenance  Information  System 

ISD 

ITI-ALC  Server  Device 

ISN 

ITI-ALC  Server  Network 

ITI-ALC 

Integrated  Technical  Information  for  the  Air  Logistics  Centers 

IWD 

ITI-ALC  Workstation  Device 

JCALS 

Joint  Continuous  Acquisition  and  Life-Cycle  Support 

JLSC 

Joint  Logistics  Systems  Center 

LAN 

Local  Area  Network 

MDC 

Maintenance  Data  Collection 

MDS 

Mission,  Design,  and  Series 
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Acronym 

Abbreviation 

Definition 

MMD 

Mobile  Management  Device 

MMSS 

Materiel  Management  Standard  System 

MOA 

Memorandum  of  Agreement 

MRRB 

Maintenance  Requirement  Review  Board 

MSD 

Maintenance  Support  Device 

MWN 

Maintenance  Wireless  Network 

0-level 

Organizational  level 

OC-ALC 

Oklahoma  City-Air  Logistics  Center 

00-ALC 

Ogden-Air  Logistics  Center 

OSD 

Organization  Status  Display 

PAC 

Production  Acceptance  Certification  System 

PCA 

Physical  Configuration  Audit 

PDM 

Programmed  Depot  Maintenance 

PDMSS 

Programmed  Depot  Maintenance  Scheduling  System 

PERT 

Program  Evaluation  Review  Technique 

PIP 

Process  Improvement  Proposal 

PM 

Process  Model 

PWPS 

Project  Workload  Planning  System 

QA 

Quality  Assurance 

QAP 

Quality  Assurance  Plan 

QAO 

Quality  Assurance  Officer  (for  ITI-ALC:  Dr.  Walt  Seward) 

RCSD 

Responsibility  Center  Status  Display 

REMIS 

Reliability  and  Maintainability  Information  System 

REQ 

Requirement 

RTC 

Requirements  Traceability  Component 

SA-ALC 

San  Antonio-Air  Logistics  Center 

SDR 

System  Design  Review 

SEI 

Software  Engineering  Institute 

S&IO 

Support  and  Industrial  Operations 

SM 

System  Model 
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Acronym 

Abbreviation 

Definition 

SM-ALC 

Sacramento- Air  Logistics  Center 

SOW 

Statement  of  Work 

SQL 

Structured  Query  Language 

SRA 

Systems  Research  and  Applications 

SSR 

Software  Specification  Review 

SSDD 

System/Segment  Design  Document 

SSS 

System/Segment  Specification 

SW 

Software 

TAFIM 

Technical  Architecture  Framework  for  Information  Management 

TIMA 

Tool  Inventory  and  Management  Application 

TLSP 

Transport  Layer  Security  Protocol 

TO 

Technical  Order 

TODO 

Technical  Order  Distribution  Office 

TRR 

Test  Readiness  Review 

VAF 

Value  Adjustment  Factor 

WBS 

Work  Breakdown  Structure 

WCD 

Work  Control  Document 

WPAFB 

Wright-Patterson  Air  Force  Base 

WR-ALC 

Warner  Robins-Air  Logistics  Center 

WSD 

Work  Status  Device 
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APPENDIXC.  SUMMARY  OF  POSSIBLE  RECOMMENDATIONS  AND 

LESSONS  LEARNED 


C.l  GENERAL  PROJECT  (REF.;  SECTION  1.0) 

Important  issues  to  be  considered  for  all  or  most  of  the  project. 

•  SigniHcant  preparation  for  the  data  collection  trips  is  critical. 

User  acceptance  is  critical  to  any  system  development  effort.  Because  the  data  collection 
trips  were  the  first  technical  interface  with  the  users  and  because  humans  have  a  tendency  to 
let  "first  impressions  be  lasting  impressions,"  the  data  collection  trips  provided  the  first  and 
best  opportunity  to  gain  the  users'  acceptance  of  and  involvement  in  the  ITI-ALC  program. 
The  data  collection  trips  were  well  planned  as  determined  by  the  positive  feedback  and 
involvement  received  from  the  ALC  personnel. 

•  The  model  development  order  must  be  such  that  the  benefits  of  their  integration  is  fully 
utilized. 

The  methodology  applied  to  the  ITI-ALC  program  was  based  on  the  developrnent  and 
analysis  of  an  integrated  set  of  models  and  reports  documenting  the  requirements, 
specifications,  design,  and  benefits  predicted  from  implementing  the  improved  process  and 
the  ITI-ALC  system.  The  models  developed  for  this  methodology  were  the  following: 

-  “AS-IS”  and  “TO-BE”  Functional  Models, 

-  “AS-IS”  and  “TO-BE”  Data  Models, 

-  “AS-IS”  and  “TO-BE”  Process  Models, 

-  “AS-IS”  and  “TO-BE”  Simulation  Models,  and 

-  System  Model. 

The  reports  developed  were  the  ITI-ALC  Business  Case,  System/Segment  Specification 
(SSS),  and  the  System/Segment  Design  Document  (SSDD).  The  development  order  of  these 
items  was  good,  but  minor  adjustments  would  have  improved  the  effective  performance  of 
the  ITI-ALC  program.  Of  specific  importance  was  the  development  and  analysis  of  the 
process  flow  and  simulation  models.  Having  established  the  requirements  for  these  models 
and  analyses  earlier  in  the  program  would  have  reduced  some  of  the  duplicative  data 
collection  efforts. 

•  Validation  difficulty  varied  among  the  various  artifacts  developed. 

The  various  "AS-IS"  and  "TO-BE"  models,  which  formed  the  foundation  for  the  program, 
required  significant  time  and  effort  for  validation  by  the  functional  experts  and  users.  These 
validation  efforts  were  implemented  using  the  readership  cycle  and  on-site  walkthroughs. 
While  the  review  results  were  sufficient  for  validation,  the  validation  of  the  SSS  handled  as  a 
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workshop  away  from  the  ALC  worksite  proved  to  be  more  productive.  Therefore,  the  use  of 
similar  workshops  for  the  "AS-IS"  and  “TO-BE”  models  would  be  recommended. 

•  Data  flow  diagrams  are  an  excellent  way  to  represent  the  system  model. 

The  top-down  approach  supported  by  data  flow  diagrams  provided  gradual,  controlled 
concept  of  operation  for  the  ITI-ALC  system.  This  approach  also  provided  a  direct  link  of 
information  from  the  functional  and  data  models,  thereby,  supporting  the  requirement  for  the 
traceability  of  information  through  the  models.  While  the  concept  of  using  data  flow 
diagrams  was  effective,  the  System  Architecture  tool  used  to  developed  these  diagrams  had 
some  short  comings  that  need  to  be  corrected  to  support  data  flow  diagram  development. 

•  The  concept  of  developing  and  using  process  flow  diagrams  via  the  Integrated 
DEFinition  (IDEF3)  notation  and  simulation  via  WITNESS  was  value  added  to  the 
overall  ITI-ALC  program. 

The  development  of  the  process  flow  diagrams  was  facilitated  by  the  existence  of  the 
functional  models  while  the  development  of  the  process  flow  models  helped  to  increase  the 
accuracy  and  completeness  of  the  functional  models.  The  process  model  provided  an 
effective  structure  to  identify  and  collect  the  performance  information  required  from 
simulation  and  provided  an  effective  structure  for  developing  and  depicting  the  simulations. 

•  The  tools  used  to  implement  the  process  flow  model  and  the  translation  to  simulation 
are  not  yet  mature. 

The  translation  capability  from  ProSim™  to  WITNESS®  was  not  complete.  The  transfer 
from  ProSim  to  WITNESS  required  a  significant  amount  of  data  addition  and  manipulation 
in  WITNESS  before  the  process  flow  could  be  exercised  using  WITNESS.  Duririg  the  ITI- 
ALC  program,  this  ineffective  transfer  of  information  was  overcome  by  developing  an  in- 
depth  understanding  of  the  transfer  requirements  and  accomplishing  much  of  it  manually. 

The  translation  from  the  process  description  to  the  simulation  was  one-directional.  The 
intent  of  IDEF3,  as  implemented  via  ProSim,  is  to  collect  process  flow  and  perform^ce 
information  so  as  to  facilitate  the  development  and  execution  of  a  simulation  model  within 
WITNESS.  However,  because  significant  effort  was  required  to  complete  the  simulation 
model  in  WITNESS,  the  value  of  the  IDEF3  model  via  ProSim  was  significantly  reduced 
once  the  first  translation  is  accomplished.  During  the  ITI-ALC  program,  this  situation  was 
addressed  by  maintaining  the  IDEF3/ProSim  representation  for  display  of  the  network  while 
adjusting  the  model  within  WITNESS  as  needed  for  the  simulation. 

The  process  model  notation,  as  represented  by  ProSim,  was  not  an  effective  presentation 
vehicle  for  two  reasons.  First,  the  information  presented  on  a  process  flow  network  was  very 
limited,  malting  it  a  labor  intensive  effort  to  read  and  analyze  a  process  flow.  Second,  the 
amoimt  of  readable  information  printed  on  one  page  was  limited,  forcing  the  model  to  be 
developed  in  short  segments.  Connecting  the  short  segments  to  form  and  effectively  depict  a 
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larger  process  was  difficult.  This  difficulty  was  reduced  by  using  the  functional  model  and 
performance  data  sheets  to  supplement  the  use  of  the  process  flow  represented  via  ProSim. 

•  An  integrated  team  effort  is  required  to  maximize  the  quality  of  the  BPIs  and  the 
products  developed  throughout  the  program. 

Throughout  the  performance  of  the  ITI-ALC  program  conununication  among  the  team 
members  was  important  to  ensure  a  unified  understanding  of  the  "  AS-IS"  and  TO-BE  depot 
maintenance  processes,  and  was  critical  during  the  development  of  the  System  Model  (SM), 
SSS,  and  SSDD.  Even  though  the  SM,  SSS,  and  SSDD  developments  are  based  on  the  "TO- 
BE"  models,  the  inteipretation  of  the  models  can  vary  slightly,  causing  potential 
inconsistencies  among  these  artifacts.  Close  interactions  among  the  developers  of  all  the 
program  artifacts,  and  especially  the  SM,  SSS,  and  SSDD  maximized  the  effectiveness  of  the 
ITI-ALC  system  requirements,  concept  of  operation,  and  high-level  design. 

C.2  "AS-IS"  FM  POSSIBLE  RECOMMENDATIONS  AND  LESSONS  LEARNED 
(REF.;  SECTION  3.1.1.4) 

The  following  possible  recommendations  or  lessons  were  learned  during  the  data  collection  and 
the  development  and  use  of  the  "AS-IS"  FM. 

•  Significant  preparation  for  the  data  collection  trips  was  critical. 

User  acceptance  is  critical  to  any  system  development  effort.  Because  the  data  collection 
trips  were  the  first  technical  interface  with  the  users  and  because  humans  have  a  tendency  to 
let  "first  impressions  be  lasting  impressions",  the  data  collection  trips  provided  the  first  and 
best  opportunity  to  gain  the  users'  acceptance  of  and  involvement  in  the  ITI-ALC  program. 
To  acquire  the  desired  positive  impression,  the  data  collection  trips  must  be  well  planned  and 
effectively  implemented. 

The  data  collection  for  the  ITI-ALC  program  was  effective.  The  strawman  model  provided  a 
solid  baseline  for  identifying  the  data  to  be  collected,  identifying  the  sources  for  the  data, 
providing  a  good  vehicle  for  training  the  interviewees  about  the  process,  and  understanding 
where  each  interviewee  fit  into  the  depot  maintenance  process  prior  to  starting  an  interview. 

The  question  set  provided  an  effective  means  for  supporting  the  data  collection  trips.  Wlule 
used  only  to  a  limited  extent  during  the  actual  interviews,  the  primary  value  of  the  question 
set  was  in  training  the  interviewers  prior  to  the  data  collection  trips.  Much  like  the  notes 
developed  and  used  by  an  orator  to  prepare  and  practice  the  speech  but  used  only  as  a  quick 
reference  during  the  delivery  of  the  speech,  the  primary  value  of  the  question  set  was  received 
during  the  preparation  of  the  data  collection  effort. 
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Maintain  a  list  of  documents  collected,  user  identified  problems  and  improvement  ideas, 
and  a  list  of  activities  already  defined. 


To  implement  an  effective  and  efficient  interview,  each  interview  should  confirm  as  well  as 
build  upon  information  from  previous  information.  To  support  this  information  building 
process,  an  up-to-date  list  of  key  information  should  be  maintained  as  reference  and  to 
reduce  unnecessary  effort. 

A  valuable  source  of  information  are  documents,  forms,  etc.  that  are  used  within  the  process 
being  described.  Listing  these  documents,  along  with  a  short  description  of  each  provides  an 
effective  way  of  understanding  what  the  interviewee  is  meaning  when  he  indicates  one  of 
these  documents  and  also  provides  a  way  of  tracking  which  documents  have  already  been 
requested  or  received. 

In  a  similar  manner,  maintaining  a  list  of  process  problems  and  solutions  provided  during 
previous  intervals  provided  a  guide  to  confirm  the  same  ideas  fi'om  multiple  sources  and  to 
build  upon  the  ideas  presented. 

A  list  of  activities  performed  within  the  process  was  also  maintained  in  the  DCD  as  a  point  of 
reference.  This  list  provides  a  means  of  learning  the  terminology,  of  mappmg  previous 
interview  information  with  information  currently  being  collected,  and  identifying  similar 
functions  identified  by  different  terminology.  Using  this  approach  produces  a  controlled  but 
not  restricted  list  of  standardized  activity  identifiers. 

•  The  data  collection  interview  must  be  carefully  controlled  by  the  interviewer. 

Interview  control  must  be  balanced  between  strict  control  where  the  interviewee  is  answering 
very  specific  questions,  and  loose  control  where  the  interviewee  is  doing  all  the  talking. 
Asking  questions  based  very  tightly  on  the  question  set  tended  to  limit  the  information 
provided  by  the  interviewee  and  assumed  that  the  interviewer's  prior  underst^ding  of  the 
process  was  accurate  because  the  interviewer  was  directing  the  information  being  obtained. 
Initiating  the  interview  with  "tell  me  what  you  do"  does  not  focus  the  interviewee  so  they 
tend  to  ramble,  hoping  they  say  something  that  is  of  importance  to  the  interviewer. 


The  most  effective  approach  was  to  explain  our  current  understanding  of  the  depot 
maintenance  process,  describe  the  types  of  information  desired,  and  then  ask  questions  that 
focused  the  interviewees  discussion  around  the  information  needed.  It  was  the  interviewers 
responsibility  to  evaluate  the  usefulness  of  the  information  being  provided  and  determine 
when  additional  questions  were  needed  to  refocus  the  discussion  to  ensure  that  the  data 
requirement  of  the  question  set  were  satisfied.  By  allowing  the  interviewee  to  describe  the 
process  in  their  terms  reduced  their  need  to  adjust  to  an  unfamiliar  way  of  discussing  their 
process.  As  a  general  rule  of  thumb,  it  was  effective  at  2  to  3  minute  intervals  to  ask  a 
refocusing  question  or  provide  some  type  of  indication  to  the  interviewee  that  appropriate 
information  was  being  provided. 


88 


Scheduling  the  interview  for  the  one-hour  duration  was  sufficient  to  collect  significant 
information  and  not  lose  the  attention  of  the  interviewee.  To  aid  in  accurately  documenting 
the  collected  information,  a  one-hour  period  should  also  be  scheduled  immediately  following 
the  interview  to  formally  record  the  information  while  it  is  still  fresh. 

•  An  end-of-day  meeting  should  be  held  during  data  collection  trips. 

The  data  collection  teams  should  meet  at  the  end  of  each  day  to  review  the  events  of  the  day 
and  to  distribute  information  among  the  team  members.  This  review  should  address  the 
individuals  interviewed  during  the  day,  a  summarization  of  the  information  obtained, 
identification  of  missing  information,  and  a  review  of  the  reference  list  containing  the 
documents  collected,  systems  and  activities  identified,  and  process  problems  and 
improvement  ideas.  The  schedule  for  the  next  set  of  interviews  should  also  be  reviewed  and 
coordinated  to  ensure  that  each  interview  will  be  as  productive  as  possible. 

•  Allowing  analysis  time  between  data  collection  trips  facilitates  the  collection  of  accurate 
and  complete  data. 

Obtaining  accurate  and  complete  data  requires  that  the  interviewee  and  interviewer  have  the 
same  interpretation  of  the  questions  being  asked  during  data  collection,  that  the  interviewee  is 
providing  accurate  information,  and  that  the  interviewer  is  interpreting  the  answers  correctly. 

The  best  way  to  identify  discrepancies  in  the  collected  data  is  to  obtain  and  analyze 
information  related  to  the  same  aspect  of  the  process  but  from  different  sources.  This 
analysis  is  facilitated  by  allowing  sufficient  time  between  the  interview  trips.  This  analysis 
verifies  the  process  imderstanding  by  the  interviewers,  verifies  consistency  among  the  data, 
and  identifies  holes  in  the  data. 

•  Effective  scoping  of  the  functional  model  is  important  to  maximize  the  benefits  received 
from  a  BPR  effort. 

The  focal  point  of  the  ITI-ALC  program  was  the  PDM  mechanics  within  the  Air  Force’s  Air 
Logistics  Centers.  However,  the  scope  was  expanded  at  program  initiation  to  include  within 
the  scope  of  ITI-ALC's  BPR  analysis  the  operational  environments  impacting  the  mechanic's 
performance  effectiveness.  This  expanded  scope  provided  the  foundation  needed  to 
effectively  streamline  the  mechanic  process  since  much  of  the  work  performed  by  the 
mechanic  was  a  duplication  or  continuation  of  the  work  performed  by  other  individuals. 
Therefore,  restricting  the  scope  of  the  ITI-ALC  program  just  to  the  mechanic  would  have 
limit  the  BPIs  identified  for  the  mechanic  and  would  have  reduced  improved  effectiveness 
provided  by  the  proposed  ITI-ALC  system. 
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C3  “AS-IS”  DM  POSSIBLE  RECOMMENDATIONS  AND  LESSONS  LEARNED 
(REF.;  SECTION  3.1.2.4) 

The  following  possible  recommendations  and  lessons  were  learned  during  the  development  and 
use  of  the  "AS-IS"  DM. 

•  Developing  the  DM  in  conjunction  with  the  FM  resulted  in  a  tight  linkage  between  the 
models. 

The  original  plan  expected  that  modeling  the  functions  performed  would  uncover  many  of  the 
requirements.  However,  we  found  that  the  data  modeling  also  uncovered  additional 
functional  requirements.  This  two-way  interchange  could  t^e  place  by  walking  through 
scenarios  in  the  FM  and  identifying  the  information  within  the  DM  that  supports  the 
scenarios.  Because  of  the  tight  linkage  between  the  DM  and  the  FM,  the  merging  of  the  two 
glossaries  was  feasible  and  more  useful  than  the  presentation  of  individual  glossaries. 

.  The  use  of  Design/IDEF  as  the  IDEFix  modeling  tool  required  some  other  conventions 
be  adopted. 

DoD  8320.1-M-l  mandates  that  a  data  element  name  consists  of  a  prime  word  name  with  its 
modifiers  and  the  generic  element  name  with  its  modifiers.  This  convention  was  not  adopted 
for  two  reasons.  The  tool  automatically  draws  the  entity  boxes  large  enough  to  accommodate 
the  largest  entity  or  attribute  name.  Using  fulling  qualified  names  made  the  diagram  much 
too  crowded.  Even  using  only  the  prime  word  name  had  to  be  relaxed  because  the  tool 
requires  that  each  attribute  witto  the  model  be  unique.  So  in  some  cases,  the  attribute  name 
was  concatenated  on  the  entity  name  to  form  a  unique  attribute  name.  For  example,  more 
than  one  entity  has  an  identifier  attribute.  To  keep  them  umque,  they  became  facility- 
identifier,  cost-center-identifier,  etc. 

•  The  data  models  could  not  be  reviewed  effectively  as  a  whole. 

The  data  models  representing  the  "AS-IS"  and  "TO-BE"  depot  maintenance  process  were  too 
large  and  complex  to  be  effectively  presented  and  reviewed  as  a  single,  complete  model.  To 
facilitate  the  reviews  and  and  model  discussions,  the  model  was  divided  into  subject  areas, 
with  subject  area  containing  groupings  of  related  information.  These  subject  areas  were 
then  same  enough  to  support  and  effective  and  focused  review.  In  addition,  the  subject  areas 
also  allowed  for  the  use  of  an  effective  model  documenation  format.  This  format  was  a  page- 
pair  structure  consisting  of  a  subject  area  diagram  and  the  associated  text  describing  the 

diagram. 

•  The  normality  rules  had  to  be  applied  in  a  manner  that  maximized  the  development 
and  usefulness  of  the  "AS-IS"  DM. 

A  primary  objective  of  data  modeling  is  to  describe  system  data  to  the  level  where  the  data  is 
completely  normalized  at  the  atomic  level  without  duplication  of  null  attribute  values.  While 
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this  is  an  optimal  goal  to  strive  for,  a  trade-off  must  sometimes  be  made  between  the  optimal 
data  model  and  a  useful  data  model.  For  the  ITI-ALC  program,  this  optimized  trade-off  was 
defined  by  the  following  conditions. 

1.  The  model  would  be  developed  to  the  3rd  normal  form.  The  5th  norm^  form  w^  not 
implemented  since  recursive  relationships  would  be  allowed  unless  additional  attributes 
needed  to  be  maintained  in  a  resolving  associative  entity.  The  4th  normal  form  wp  not 
implemented  to  allow  for  the  use  of  null  attribute  values.  This  avoided  the  creation  of 
many  category  entities  needed  to  eliminate  null  attributes  but  decreased  the  ease  of 
understanding  the  data  contained  in  the  model. 

2.  Not  all  data  was  broken  down  to  its  atomic  or  normalized  data  elements.  Trying  to 
achieve  all  the  atomic  data  elements  involved  significant  effort  but  provided  minimal 
benefit  return.  Also,  representing  the  atomic  data  elements  would  significantly  reduce  the 
users'  understanding  of  the  model  since  they  do  not  directly  use  or  reference  the  atomic 
data  elements. 

C.4  “AS-IS”  PM  AND  SIMULATIONS  POSSIBLE  RECOMMENDATIONS  AND 
LESSONS  LEARNED  (REF.;  SECTION  3.13.4  &  3.2.3.4) 

The  requirement  to  use  IDEF3  and  simulation  analysis  within  the  ITI-ALC  program  was  (1)  to 
provide  a  test  case  for  using  the  IDEF3  and  simulation  techniques,  and  (2)  to  evaluate  ^d  use 
these  performance  analysis  capabilities  to  produce  improved  requirements  and  specifications  for 
the  streamlined  depot  maintenance  process  and  the  ITI-ALC  system.  With  respect  to  these 
objectives,  the  following  are  possible  recommendations  and/or  lessons  that  were  learned. 

•  Simulation  is  worth  the  effort. 

The  concept  of  applying  IDEF3  and  WITNESS  to  a  BPR  effort  provides  significant  benefits. 
Using  the  tools  as  stated  by  the  IDEF3  specifications  and  obtaining  these  benefits  m  total  was 
a  challenge.  However,  based  on  this  application  experience  and  the  specifications  for  IDEF3 
and  ProSim  specifically,  the  following  recommendations  and/or  lessons  learned  resulted  from 
developing  the  "AS-IS"  PM  and  the  corresponding  simulation. 

•  ProSim  did  not  implement  the  IDEF3  rules  completely  as  they  were  specified  in 
Information  Integration  for  Concurrent  Engineering  (IICE)  IDEF3  Process  Description 
Capture  Method,  dated  May  1992. 

The  IDEF3  specification  identifies  five  object  states  available  throughout  the  project.  The 
Entity  Description  Type  is  a  pooled  item  in  ProSim  and  can  only  be  set  to  one  type  for  the 
entire  project.  The  IDEF3  specification  identifies  junctions  as  providing  a  niechanism  to 
specify  the  logic  of  process  branching.  ProSim  junctions  expand  beyorid  branching  logic  ^d 
contain  much  of  the  fimctionality  required  by  processes,  such  as  creating  objects  or  passmg 
multiple  created  objects. 
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•  The  translation  from  the  process  description  to  the  simulation  was  one-directional. 

The  intent  of  IDEF3,  and  implemented  via  ProSim,  is  to  collect  process  flow  and  performance 
information  so  as  to  facilitate  the  development  and  execution  of  a  simulation  model  within 
WITNESS.  Following  the  translation  from  ProSim  to  WITNESS,  a  significant  amount  of 
effort  is  required  to  adjust  and  enhance  the  information  in  WITNESS  before  the  simulation 
model  is  operational  and  available  for  verification,  validation,  and  analysis. 

Because  of  this  additional  work  required  within  WITNESS,  and  because  the  adjustments 
made  in  WITNESS  can  not  be  transferred  back  to  ProSim,  the  benefits  of  the  IDEFa/ProSim 
representation  and  capabilities  are  lost  once  the  first  translation  occurs. 

During  the  ITI-ALC  program,  this  situation  was  addressed  by  maintaining  two  separate  but 
correlated  models.  The  IDEFa/ProSim  representation  was  maintained  for  display  of  the 
network  while  the  WITNESS  model  was  maintained  for  exercising  the  simulation. 

•  The  process  model  notation,  as  represented  by  ProSim,  was  not  an  effective 
presentation  vehicle. 

The  information  represented  on  a  process  flow  network  is  very  limited,  reducing  its 
effectiveness  as  a  communication  and  analysis  tool.  To  read  and  understand  a  network 
requires  a  labor  intensive  process  of  obtaining  and  integrating  information  from  other 
sources. 

The  readability  of  an  IDEF3  network  could  be  significantly  increased  if  the  process  flow 
information  could  be  overlaid  onto  the  network.  For  example,  labels  should  be  placed  on  the 
relationship  arrows,  conditions  listed  with  the  branches,  and  timing  and  resource 
requirements  placed  on  processes. 

To  support  the  presentation  and  readability  of  the  IDEF3  process  flows  for  the  ITI-ALC 
program,  the  functional  models  and  performance  data  worksheets  were  used  to  discuss  and 
validate  the  process  flows.  This  approach  improved  the  communications,  but  using  just  the 
performance  worksheets  proved  to  be  the  most  effective  for  discussing  and  validating  the 

process  flows. 

•  Scenarios,  or  small  snippets  of  the  entire  process,  help  support  the  presentation  of  the 
process,  but  did  not  provide  the  foundation  necessary  to  effectively  analyze  the 
performance  of  the  depot  maintenance  process. 

The  IDEF3  concept,  and  specifically  the  ProSim  tool,  was  developed  b^ed  on  the  idea  that  a 
large  process  flow  could  be  developed  and  analyzed  as  small  scenarios  or  snippets  of  the 
entire  process.  These  small  scenarios,  consisting  of  five  to  ten  processes,  could  then  be 
presented  on  a  single  page. 
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However,  this  segmented  approach  limits  the  capability  to  evaluate  the  impacts  caused  by 
changes  across  the  entire  process  flow.  Evaluating  the  entire  process  as  a  umt  was  especially 
important  when  trying  to  identify  operational  bottlenecks  when  parameters  in  one  segment 
were  changed. 

•  The  data  collection  for  the  "AS-IS"  PM  and  the  simulation  would  have  been 
approached  differently  if  this  modeling  and  analysis  requirement  had  been  established 
at  the  beginning  of  the  ITI-ALC  program. 

The  performance  information  required  to  develop  the  "AS-IS"  PM  would  have  been  collected 
as  part  of  the  data  collection  for  the  "AS-IS"  FM. 

C.5  “TO-BE  FM  POSSIBLE  RECOMMENDATIONS  AND  LESSONS  LEARNED 

(REF.;  SECTION  3.2.1.4) 

The  following  possible  recommendations  and/or  lessons  were  learned  during  the  development 

and  use  of  the  "TO-BE"  FM. 

•  Effective  scoping  of  the  functional  model  is  important  to  maximize  the  benefits  received 
from  a  BPR  effort. 

The  focal  point  of  the  ITI-ALC  program  was  the  PDM  mechanics  within  the  Air  Force's  Air 
Logistics  Centers.  However,  the  scope  was  expanded  at  program  initiation  to  include  witMn 
the  scope  of  ITI-ALC’s  BPR  analysis  the  operational  environments  impacting  the  mechanic's 
performance  effectiveness.  This  expanded  scope  provided  the  foundation  needed  to 
effectively  streamline  the  mechanic  process  since  much  of  the  work  perfonned  by  the 
mechanic  was  a  duplication  or  continuation  of  the  work  performed  by  other  individuals. 
Therefore,  restricting  the  scope  of  the  ITI-ALC  program  just  to  the  mechamc  would  have 
limit  the  BPIs  identified  for  the  mechanic  and  would  have  reduced  improved  effectiveness 
provided  by  the  proposed  ITI-ALC  system. 

•  By  including  the  cursory  look  at  the  engine  and  component  environments  allowed  for 
the  understanding  for  how  the  ITI-ALC  system  could  aid  the  entire  depot  maintenance 
process. 

While  the  focus  of  the  ITI-ALC  program  was  the  PDM  mechanic,  the  basic  activities 
performed  by  all  personnel  throughout  the  depot  have  much  in  common  and  ^e  tightly 
integrated.  Making  the  minimal  effort  required  to  imderstand  a  larger  perspective  of  the 
depot  operations  in  terms  of  the  commonality  and  integration  provided  the  ITI-ALC  team 
with  the  necessary  backgroimd  to  maximize  Ae  benefits  received  from  the  improved  depot 
maintenance  process  and  the  implementation  of  the  ITI-ALC  system.  Therefore,  using  this 
broader  perspective  assured  that  the  improvement  concepts  addressed  all  common  aspects 
and  assured  that  recommended  improvements  did  not  negatively  impact  other  processes  and 
personnel  involved  in  depot  maintenance. 
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C.6  “TO-BE”  DM  POSSIBLE  RECOMMENDATIONS  AND  LESSONS  LEARNED 
(REF.;  SECTION  322.4) 

The  following  possible  recommendations  and/or  lessons  were  identified  during  the  development 
and  use  of  the  “TO-BE”  DM. 

•  The  ERWin  tool  had  significant  limitations  with  respect  to  producing  quality  looking 
documentation. 

Models  are  developed  for  use  as  analysis  tools,  therefore,  they  must  be  documented  and 
presented  in  a  manner  that  facilitates  the  analysis  process.  While  the  ERWin  tool  facilitated 
the  development  by  controlling  the  migration  of  attributes  among  the  entities,  the  tool  did  not 
allow  for  user  control  of  the  relationship  layout  among  the  entities.  This  resulted  in  <^ta 
model  that  was  difficult  and  time  consuming  to  read  and  analyze.  To  correct  this  situation, 
the  completed  ERWin  model  was  exported  to  the  Design/IDEF  tool  which  allowed  the 
developer  to  rearrange  the  entities  and  relationships  so  as  to  maximize  the  readability  and 
usefulness  of  the  model. 

C.7  SM  POSSIBLE  RECOMMENDATIONS  AND  LESSONS  LEARNED  (REF.; 
SECTION  3.3.4) 

During  the  development  of  the  SM  model,  the  following  possible  recommendations  and/or 
lessons  learned  were  identified. 

•  Data  flow  diagrams  are  an  excellent  way  to  represent  the  system  model. 

The  goal  of  the  system  model  is  to  provide  a  transition  firom  the  functional  and  data 
requirements  identified  via  the  functional  and  data  models  to  the  design  of  the  system  that 
will  satisfy  those  requirements.  Using  data  flow  diagrams  provided  an  effective  way  of 
presenting  this  transition. 

The  data  flow  diagrams  supported  a  top-down  approach  and  provided  a  leveling  of 
information  that  reduced  the  complexity  of  information  on  any  one  diagram  while  allowing 
for  increasingly  more  detailed  information  on  decomposed  diagrams.  This  top-down 
approach  corresponded  to  that  used  to  developed  the  functional  model,  resulting  in  a  strong 
correlation  between  the  'TO-BE"  functional  model  and  the  system  model. 

The  data  flow  diagrams  accommodated  the  specification  of  data  stores  and  the  interfaces  with 
the  data  stores.  The  defimtion  of  these  data  stores  provided  an  effective  link  to  the  entities 
specified  within  the  "TO-BE"  data  model. 

The  flow  diagrarns  accommodated  the  incorporation  of  system  control  activities  and  the 
specification  of  control  flows  among  the  activities  and  the  interfaces  to  the  user,  the  internal 
data  stores,  and  the  external  database  systems.  Furthermore,  a  side  benefit  of  this  operate  is 
that  superior  cost  estimates  based  on  Function  Points  can  be  derived  from  the  models. 
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•  Although  DFDs  are  an  excellent  way  to  represent  the  system  model,  the  full  benefit  of 
the  model  can  not  be  realized  by  those  not  knowledgeable  in  the  model  language. 

Non-native  languages  have  advantages  over  a  native  language  because  they  are  developed  to 
efficiently  present  specific  types  of  information.  Native  languages,  on  the  other  hand,  have 
the  advantage  in  that  they  are  common  and  understood  by  a  majority  of  people,  even  though 
they  may  not  present  the  information  as  efficiently  and  accurately.  This  language  difference 
causes  some  problems  because  the  model  developers  prefer  the  non-native  languages  while 
system  users  often  prefer  the  native  languages.  To  maximize  the  benefit  of  these  two 
perspectives,  a  two  step  approach  should  be  used  that  enhances  the  communications  between 
the  two  situation.  First,  users  must  be  motivated  and  willing  to  take  the  effort  to  gain  a 
reasonable  understanding  of  the  non-native  language  in  order  to  fully  appreciate  the 
information  contained  in  the  model  and  to  fully  utilize  the  model.  Second,  the  developers 
must  be  motivated  and  willing  to  supplement  the  non-native  language  with  an  ICON  level 
abstraction  to  facilitate  the  understanding  by  the  users.  Development  of  a  common 
understanding  and  effective  communication  maximizes  the  effectiveness  and  usefulness  of 
the  information  produced. 

C.8  BUSINESS  CASE  POSSIBLE  RECOMMENDATIONS  AND  LESSONS  LEARNED 

(REF.;  SECTION  3.4.4) 

During  the  development  of  the  business  case,  the  following  possible  recommendations  and/or 

lessons  learned  were  identified. 

•  Domain  objective  were  not  available. 

There  were  no  AFMC-wide  objectives  dealing  explicitly  with  the  ITI-ALC  domain.  As  a 
result,  the  ITI-ALC  team  inferred  two  specific  priority  objectives  from  the  many  general 
objectives  included  in  AFMC  and  ALC  planning  documents.  Those  specific  objectives  were 
reduced  operating  expense  and  aircraft  flow  days  for  orgamc  aircraft  PDM,  as  discussed  in 
the  business  case. 

Also,  there  were  no  command-wide  performance  measures  and  targets  (metrics)  dealing  with 
the  domain  that  ITI-ALC  was  intended  to  affect.  Accordingly  the  team  inferred  measures  and 
targets  from  correspondence  and  data  available  within  the  depot  maintenance  domain. 

•  Process  improvement  proposal  structure  required  more  than  one  improvement 
alterative. 

This  project  team  recognized  that  previous  Information  Technology  (IT)  investment  projects 
typically  presented  decision  makers  with  one  alternative  for  improvement  investment 
decisions.  Therefore  early  on  the  ITI-ALC  project  incorporated  an  initiative  to  present  a 
series  of  more  than  one  option  for  the  decision-maker.  Each  option  more  aggressive  than  the 
previous  option  in  the  series. 
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•  Two-way-blind  test  could  not  be  fully  implemented. 

This  project  recognized  early  in  the  program  that  there  were  not  enou^  “equal” 
resources  to  conduct  a  true  two-way  blind  test.  One  solution  to  this  issue  is  to  estimate 
enough  resources  in  the  project  cost  estimate.  This  is  very  expensive  and  may  not  be  possible 
if  equal  members  cannot  be  found.  If  equal  team  members  cannot  be  found  a  control 
tpam  would  also  have  to  be  formed  to  allow  for  the  evaluation  of  variables  between  the  two 

teams.  In  this  age  of  “do  more  with  less”,  this  is  not  an  acceptable  approach.  Using  two  core 

teams  with  one  or  two  team  members  in  both  teams  worked  as  long  as  everyone  excepts  the 
condition  that  the  learning  effect  may  bias  the  results.  To  minimize  this  bias,  different  team 
leaders  were  used  and  the  SME  was  kept  as  “blind  as  possible”  to  the  overall  process  so  as 
not  to  subconsciously  direct  the  two  processes  so  they  artificially  converge.  T^s  plus  an 
independent  validation  on  the  simulation  half  of  the  experiment  (see  the  validation  section 
above),  allowed  for  results  with  small  confidence  intervals. 

•  Another  measure  for  economic  analysis. 

DoD  and  Air  Force  directives  on  economic  analysis  and  the  development  of  business  cases 
contain  two  views  of  economic  analysis. 

In  the  ITI-ALC  project,  the  traditional  view,  defines  the  problem  with  the  question,  “what  is 
the  maintenance  cost  to  accomplish  organic  aircraft  PDM  and  how  can  we  do  it  for  less? 
The  answer  obviously  includes  the  cost  of  mechanics,  engineers,  and  managers,  &e  parts  and 
raw  materials  incorporated  into  the  aircraft  product,  the  hangers  and  oAer  facilities  and  the 
cost  of  supporting  resources.  It  might  be  termed  the  in-house  cost.  It  is  not  the  major  cost. 
This  view  with  the  problem  from  a  systems  perspective.  It  does  not  call  into  question 
the  “out-of-service”  costs  associated  with  the  relationship  between  the  customer  and  the 
provider,  AFMC.  This  problem  is  not  new.  It  was  raised  by  the  DUSD(L)  in  1995. 

In  a  May  30,  1995  memo,  the  DUSD(L)  made  a  statement  in  response  to  GAO/AIMD-95-1 10 
Depot  Maintenance  Standard  Systems.  The  statement  included,  “While  the  Department 
appreciates  the  GAO  acknowledgment  of  process  improvements  achieved  to  date  in  depot 
maintenance,  the  GAO  fails  to  realize  the  magnitude  of  achieving  significant  reductions  in 
cost  and  flow  days  equating  to  a  30%  reduction  in  the  cost  of  ship  overhaul,  or  processing 
two  additional  B-1  bombers  through  an  ALC  because  of  reductions  in  flow  days.  To 
illustrate,  due  to  the  complete  reengineering  of  work  processes  and  use  of  BAIM,  the  Navy 
has  reduced  overhaul  time  for  the  688  class  submarine  from  24  to  20  months  and  now 
estimates  all  future  688  workloads  will  take  18  months.  To  put  that  change  in  perspective, 
the  previous  average  for  similar  work  was  $81  million  per  submarine.  However,  in  addition 
to  reducing  the  cost  of  overhaul,  weapon  systems  are  expedited  to  the  warfighter,  increasing 
mission  readiness.  In  addition  fewer  systems  in  the  repmr  cycle  equates  to  fewer  systems 
needing  overall,  thereby  achieving  even  more  dramatic  savings.” 

Without  a  value  on  the  amount  of  time  a  system  is  out-of-service,  the  repair  cycle  days,  it  is 
difficult  for  managers  to  make  investments  to  reduce  them.  The  ITI-ALC  system  therefore 
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developed  a  second  view  that  combines  total  process  and  cost  views  into  a  total  systems 
view.  It  includes  the  traditional  view  and  takes  into  account  the  needs  of  and  co^s  to  the 
customer.  In  this  view,  the  question  is  not  only  “what  is  the  maintenance  cost?  ,  but  in 
addition,  “what  are  the  costs  to  the  customer?’’’' . 

What  does  the  customer  give  up  to  obtain  a  PDM  or  incorporation  of  a  modification  package 
in  an  aircraft?  A  customer  gives  up  two  things.  First,  a  customer  pays  the  maintenance  cost 
for  each  aircraft.  That  cost  is  relatively  strai^tforward  as  reflected  in  the  traditional^  view 
discussed  above.  Second,  a  customer  relinquishes  use  of  that  portion  of  the  aircraft’s  life 
spends  in  the  maintenance  process.  For  PDM  the  period  will  vary,  but  ranges  from  100  to 
several  hundreds  of  days  per  PDM  cycle. 

The  cost  to  the  customer  of  those  days  is  less  straightforward  than  the  representing  the 
maintenance  cost.  However,  it  was  developed  by  the  ITI-ALC  team,  based  on  similar 
industry  practice. 

To  develop  the  cost  to  the  customer,  the  ITI-ALC  team  obtained  the  Unit  Flyaway  Cost 
(UFC)  for  major  aircraft  in  the  USAF  fleet  from  Air  Force  Instruction  65-503.  This 
information  for  ten  aircraft  types  is  included  in  the  table  below.  These  items  are  included  in 
the  determination  of  a  unit  flyaway  cost  under  Appropriation  3010  (Aircraft  Procurement); 
airframe,  propulsion,  electronics,  avionics,  engineering  change  orders,  if  ^y,  government 
furnished  equipment,  first  destination  transportation  unless  a  separate  line  item,  system  and 
project  management  and  system  test  and  evaluation  if  funded  by  the  aircraft  procurement 
appropriation,  warranties,  recurring  costs  (both  contract  and  in-house),  nonrecurring  cost 
(both  contract  and  in-house),  and  advances  buy  cost  (see  Table  C-1). 


Table  C-l.  Aircraft  Cost  Information 


Aircraft  MDS 

AFI 65-503  UFC 
(millions  $) 

UFC  *  1.2 
(millions  $) 

Value  of  One 
Aircraft  Day  ($)^ 

B-1 

86 

240.7 

288.8 

39566 

F-15 

584 

24.2 

29 

3982 

F-16 

1588 

14.5 

17.4 

2384 

F-22 

442 

68.1 

81.7 

11197 

E-3 

29 

114.3 

137.2 

18795 

C-5 

76 

135.6 

162.7 

22300 

C-130 

877 

14.5 

17.4 

2387 

KC-135R 

400 

52.7 

63.2 

8664 

C-141 

223 

41.2 

49.5 

6785 

C-17 

120 

293.2 

351.8 

48199 

‘USAF  Fact  Sheets 

^  FY95  to  FY94  conversion  aircraft  procurement  weighted  factor  of  1 .032. 
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Unit  flyaway  cost  does  not  include:  research,  test,  and  evaluation  appropriation  expenditures, 
weapons  and  armament  (except  if  part  of  the  airframe,  e.g.,  the  30N^  GAU-81A  gun  on  &e 
A- 10),  peculiar  ground  support  equipment,  peculiar  training  equipment,  technical  data,  initial 
spares  and  replacement  spares. 

In  regards  to  flyaway  cost  and  modifications,  it  is  important  to  note  that  UFC  reflects  only 
those  modifications  which  produced  a  new  MDS.  For  example,  the  EF-1 1 1 A  was  modified 
from  the  F-lllA.  Major  aircraft  modifications  which  do  not  produce  a  new  MDS  are  not 
included.  Thus,  the  unit  flyaway  cost  for  the  B-052H  reflects  the  unit  flyaway  cost  as 
originally  produced  and  then  inflated  to  the  constant  dollars  of  a  specific  fiscal  year.  Since 
subsequent  modifications  to  the  B-052H  did  not  produce  a  new  MDS,  the  modifications  are 
not  included  in  the  unit  flyaway  cost  of  the  B-052H. 

To  account  for  research,  development,  test  and  evaluation,  techmcal  data,  support  equipment, 
etc.,  the  ITI-ALC  team  increased  UFC  by  20%.  The  ITI-ALC  team  assumed  20  years  in  an 
aircraft  life  cycle. 

To  calculate  the  value  of  one  aircraft  flow  day  the  team  applied  this  formula: 

(UFC)(1.2) 

CcstUthcCummerMmAtrcr^flFkwD^  -  cycle  in  Y«,rs)(36S  Days) 

Using  this  approach,  the  value  of  MSIP  F-15  flow  days  of  174  days,  is  $692,868  per  aircraft. 
The  value  of  the  154  F-15  flow  days  provided  to  a  customer,  if  flow  days  could  be  reduced  to 
20  days,  is  $613,226  per  aircraft.  The  ITI-ALC  team  estimated  Primary  Aircraft 
Authorization  (PAA)  fleet  sizes  from  USAF  fact  sheets  or  public  literature.  For  a  fleet  of  584 
F-15  aircraft,  the  value  of  the  returned  flow  days  is  $358,125,152  over  one  5  or  6  year  PDM 
cycle. 

C.9  SSS  POSSIBLE  RECOMMENDATIONS  AND  LESSONS  LEARNED  (REF.; 
SECTION  3.5.4) 

The  process  used  to  develop  the  ITI-ALC  SSS  was  fairly  rigorous,  systematic,  and  successful. 
There  were  several  possible  recommendations  and/or  lessons  learned  from  unplementing  this 
process. 

•  SSS  development  workshop  should  be  longer. 

The  workshop  was  three  days  long  which  only  allowed  time  for  revising  the  user 
requirements.  The  workshop  should  be  five  days  long  to  allow  time  to  examine  and  revise  all 
requirements  in  the  SSS  with  focus  on  the  user  requirements,  external  interface  requirements, 
and  segment  descriptions  and  requirements. 

•  SSS  workshop  instead  of  one  of  the  three  deliverables. 

If  a  workshop  is  used  as  a  means  to  develop  the  SSS,  there  should  only  be  two  deliveries  of 
the  document.  There  should  be  one  delivery  (preliminary  final)  reflecting  the  results  of  the 
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workshop  and  a  second  delivery  (final)  approximately  three  months  later  to  address  any 
outstanding  dptails  or  issues  resulting  from  customer  review  of  the  post-workshop  version. 
The  premise  of  two  deliveries  stems  firom  the  idea  that  the  workshop  is  supported  by  the 
appropriate  personnel  that  can  and  should  mfluence  the  requirements  within  the  SSS, 
negating  the  need  for  three  deliveries  to  achieve  the  desired  content  level. 

•  Iterative  requirements  development  is  required. 

The  primary  categories  of  developmental  requirements  within  the  SSS  are  functional 
requirements  (representing  user  requirements),  external  interface  requirements,  and  segment 
requirements.  The  functional  requirements  should  be  defined  first  and  should  determine  the 
need  and  provide  the  basis  for  the  external  interface  requirements  and  the  segment 
requirements.  As  requirements  are  developed,  each  category  (functional,  external  interface, 
and  segment)  should  be  revisited  and  revised  accordingly  (iterative  process)  until  “all” 
requirements  have  been  defined  for  the  system. 

•  The  concept  for  how  to  specify  and  organize  an  effective  SSS  was  not  consistent  among 
all  individuals  involved  in  its  development,  even  though  a  DID  was  identified  within  the 
contract. 

The  SSS  was  one  of  the  key  documents  developed  during  the  ITI-ALC  program  in  that  it 
specified  the  detailed  requirements  for  the  "TO-BE"  ITI-ALC  process  as  well  as  the 
technologies  used  to  implement  the  process.  The  true  success  of  the  ITI-ALC  program  will 
not  be  determined  until  the  SSS  is  used  as  the  guide  for  designing,  developing,  and 
implementing  the  ITI-ALC  system.  Given  the  importance  of  the  SSS,  the  “correct”  DID,  and 
a  full  understanding  of  that  DID,  should  have  been  established  early-on  by  all  representatives 
of  the  customer  and  should  have  been  reinforced  throughout  the  program.  The  reinforcement 
would  ensure  that  new  program  personnel  understand  the  history  and  goals  of  the  work 
performed  and  would  eliminate  the  possibility  of  unnecessary  program  direction  chanps. 
These  variations  in  SSS  expectations  and  concepts  within  the  customer  team  required 
significant  time  and  effort  to  discuss  and  a  large  amoimt  of  rework. 

C.IO  SSDD  POSSIBLE  RECOMMENDATIONA  AND  LESSONS  LEARNED  (REF.; 

SECTION  3.6.1.4) 

The  following  are  possible  recommendations  and/or  lessons  that  were  learned  during  the 

development  the  SSDD. 

•  Overlap/parallel  development  of  project  deliverables. 

While  overlapping  the  development  of  project  documents  such  as  the  SSDD  and  the  SSS 
helps  reduce  the  project’s  schedule,  this  overlap  also  can  cause  extra,  non-value-added  work 
when  changes  are  made,  due  to  the  ripple  effect.  The  solution  used  on  this  project  didn  t 
eliminate  this  rework  completely,  but  did  reduce  it  to  tolerable  levels.  The  primary  way  this 
was  done  was  to  structure  the  development  approach  to  take  advantage  of  the  iterative  nature 
of  the  system  development  process  (see  Figure  3-11).  The  “TO-BE”  models  represented 
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requirements  for  the  system  and  were  baselined  early  enough  in  the  process  to  be  used  to 
develop  the  SM.  The  SM  then  became  the  major  input  into  the  SSDD.  Became  both  the 
SSS  and  the  SSDD  were  ultimately  derived  from  the  same  sources,  the  SSDD  fulfilled  all  the 
requirements  identified  in  the  SSS  even  with  them  being  developed  almost  simultaneously. 
This  fact  was  validated  using  the  traceability  process  outlined  above.  This  innovative 
approach  to  parallel  development  allowed  shorter  schedule  time  as  well  as  flexibility  to 
change. 

C.11  IMIS  DEMO  ANALYSIS  POSSIBLE  RECOMMENDATIONS  AND  LESSONS 
LEARNED  (REF.;  SECTION  3.6.2A) 

During  the  performance  of  the  IMIS  Demo  Analysis,  the  following  possible  recommendations 
and/or  lessons  learned  were  identified: 

•  System  documentation  is  needed. 

Comparing  a  developed  system  with  a  system  undergoing  requirements  definition  presented  a 
unique  challenge  that  would  have  been  reduced  if  the  documentation  for  the  developed 
system  had  been  provided. 

•  Working  with  the  software  of  the  developed  system  was  very  useful  for  determining  the 
functionality  of  the  system. 

Reading  the  documentation  for  a  system  helps  imderstand  the  intended  functionality  of  the 
system  and  provides  the  mechanics  necessary  to  maneuver  within  the  system.  Having  gained 
this  fundamental  understanding,  getting  hands-on  experience  is  the  only  way  of 
understanding  the  real  system  capabilities  necessary  to  perform  an  indepth  evaluation  and 
comparison. 

•  Development  system  source  code  would  have  enhanced  the  comparison. 

The  comparison  document  could  have  provided  anofiier  level  of  usefulness  if  the  software 
code  had  been  provided.  The  code  would  have  allowed  the  identification  of  the  changes 
required  in  the  code  rather  than  the  functions  that  require  change. 

C.12  DEMONSTRATION  PLAN  POSSIBLE  RECOMMENDATIONS  AND  LESSONS 
LEARNED  (REF.;  SECTION  3.7.4) 

The  following  observations  were  made  as  possible  recommendations  to  improve  the  results  and 
efficiency  of  future  efforts  of  this  nature: 

•  The  title  of  the  resulting  document  for  this  effort  should  have  been;  Experiment  Design 
Document  for  the  Demonstration  of  the  Efficacy  of  the  ITI-ALC  System  and  BPIs. 

It  was  obvious  from  ITl-ALC’s  Statement  of  Work  (SOW)  that  AL/HRGO  wanted  a  plan  for 
an  experiment  that  demonstrates  the  improved  effectiveness  and  efficiency  of  mechanics  and 
other  depot  PDM  personnel  when  using  new  processes  and  technologies  described  in  the 
System/Segment  Specification  and  the  ITI-ALC  Business  Case.  This  intent  was  not  obvious 
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from  the  title  of  the  document  per  the  CDRL  list.  A  “demonstration”  denotes  a  certain 
informal  approach  to  the  effort  and  a  “test”  would  most  likely  be  confused  with  a  “System 
Test”  which  has  a  different  intent  (the  intent  of  all  formal  system/software  testing  is  to  find 
errors  in  the  system/software  [Rubey86])  then  what  will  be  done  for  Phase  11  of  ITI-ALC.  In 
the  future,  AL/HRGO  should  name  similar  efforts  as  “Experiments”. 

•  Adjust  the  development  schedule  for  the  Demostration  Plan. 

Before  time  and  resources  are  spent  on  designing  an  experiment,  customer  resources  should 
be  allocated  to  the  review,  analysis,  understanding  and  critiquing  of  the  results  of  that  effort. 
If  another  such  program  is  conducted  in  the  future,  AL/HRGO  should  consider  removing  this 
tasks  from  the  Phase  I  SOW  due  to  it’s  premature  nature.  This  task  is  better  suited  to  a  time 
closer  to  the  actual  implementation  of  the  plan. 

•  Cost  consideration  for  the  experiment  should  be  included. 

The  test  or  demonstrations  identified  in  the  deliverable  for  this  effort  did  not  presume  to 
evaluate  the  cost  of  performing  the  tasks  identified  in  the  experiment  or  the  cost  of  building  a 
demonstration  system  to  conduct  those  tasks.  However,  the  design  was  developed  so  that  one 
half  of  the  experiment  could  be  conducted  with  little  of  no  interaction  ivith  the  second  half, 
therefore  allowing  for  a  truncated  test/demonstration.  Another  way  to  do  this  would  be  to  get 
potential  users  involved  much  earlier  (e.g.,  at  the  second  release  of  the  SSS)  in  the  process  to 
define  which  functional  requirements  they  would  like  to  see  in  a  test/demonstration.  This 
information  would  then  have  to  be  made  available  to  the  contractor  performing  the 
experiment  design  effort. 

•  More  emphases  on  functional  demonstration  verses  system  interface. 

To  ensure  a  cost  effective  demonstration,  more  emphases  should  be  put  on  prototyping  the 
functional  capabilities  of  the  ITI-ALC  system  verses  making  something  that  will  evolve  into 
a  production  system.  One  aspect  of  “production  level”  software  is  that  too  much 
time/resources  are  spent  in  building  “real”  system-to-system  interfaces  with  legacy  and 
emerging  system.  Most  of  the  data  obtained  from  the  supporting  system  included  in  the  ITI- 
ALC  design  can  be  simulated/derived,  allowing  for  more  resources  devoted  to  building  a 
system  that  helps  the  user  understand  the  ITI-ALC  BPI  concepts  and  that  is  flexible  enough 
to  try  many  different  ideas.  The  industries  trend  toward  “evolutionary”  prototypes  is  an 
unrealized  potential  because  the  nature  of  prototypes  and  the  nature  of  “production”  software 
are  mostly  mutually  exclusive.  This  lesson  highlights  and  utilizes  the  flexibility  and 
modularity  of  the  ITI-ALC  design  per  the  SSDD  and  the  SM. 
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APPENDIX  D.  DATA  COLLECTION  APPROACH 


D.l  DATA  COLLECTION  DISCUSSION 

The  various  models  developed  as  part  of  the  ITI-ALC  program  formed  the  foundation  for 
specifying  the  improved  depot  maintenance  process  and  for  establishing  the  requirements  and 
specifications  needed  to  identify  and  select  the  technologies  capable  of  implementing  the 
improved  PDM  process.  Model  development  began  as  die  first  step  in  the  data  collection  effort 
and  guided  the  overall  data  collection  effort.  The  various  perspectives  of  the  depot  maintenance 
process  provided  by  the  models,  the  close  interaction  among  the  models,  and  the  traceability  of 
requirements  fixim  the  users  to  the  design  via  the  model  ensured  that  requirements  were 
developed  for  a  complete,  user-oriented  depot  maintenance  process  and  ITI-ALC  system. 

Figure  D-1  presents  an  overview  of  the  model  development  effort.  The  data  was  collected  fi'om 
the  users  and  formally  documented  in  the  DCD,  and  supplemented  with  notes,  collected  forms 
and  documents,  and  tape  recordings.  This  information  was  then  analyzed  and  used  to  developed 
the  set  of  "AS-IS"  and  "TO-BE"  models. 


DM  (IDEF,„)  FM  (IDEFo)  PM  (IDEF3) 


Figure  D-1.  Data  Analysis  and  Model  Development  Process 

The  "AS-IS"  FM  was  develop  by  using  the  collected  information  to  refine  and  expand  the 
strawman  model.  Using  the  name  of  a  strawman  activity  as  a  pointer  into  the  DCD,  the  details 
of  that  activity  were  selected  firom  the  DCD  via  a  variety  of  reports.  This  focused  information 
was  analyzed  and  organized  to  produce  the  decompositions  for  that  strawman  activity.  The 
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tapes,  notes,  forms,  and  the  ITI-ALC  team's  functional  experts  were  used  to  augment  the 
information  from  the  DCD. 

Once  the  Axx  level  of  the  "AS-IS"  FM  were  relatively  firm,  the  development  of  the  "AS-IS" 
DM  was  initiated.  This  development  started  with  a  listing  of  the  arrows  extracted  from  the  "AS- 
IS"  FM.  The  arrow  listing  was  supplemented  with  information  from  the  DCD,  tapes,  notes, 
forms,  and  the  ITI-ALC  team's  functional  experts.  The  "AS-IS"  models  were  not  developed 
independent  of  one  another.  Because  each  of  the  three  models  represented  the  same  process,  but 
from  a  different  perspective,  their  development  was  done  in  a  coordinated  maimer.  As  ideas, 
missing  information,  etc.  were  identified  in  one  model,  that  insight  was  transferred  to  the  other 
model  development  efforts,  the  corrective  action  taken,  and  the  resulting  information  made 
available  to  each  of  the  modeling  efforts.  This  close  coordination  helped  to  ensure  that  each  of 
the  models  was  complete  and  accurate,  and  there  was  consistency  and  traceability  among  the 
models. 

The  "AS-IS"  models  were  then  analyzed  to  identify  the  BPIs  for  the  improved  process.  These 
BPIs  were  used  to  modify  the  "AS-IS"  models  to  represent  the  improved  process  definition  via 
the  "TO-BE"  models. 

The  goal  of  the  ITI-ALC  program  was  to  analyze  the  PDM  process  within  each  of  the  five  ALCs 
and  develop  a  generic  representation  of  a  “composite”  ALC.  To  ensure  an  accurate  and 
complete  description  of  the  current  depot  maintenance  process  rather  than  a  theoretical 
perspective  of  what  the  process  should  be,  data  collection  was  performed  primarily  by 
interviewing  functional  experts  and  users,  as  opposed  to  collecting  information  solely  from 
reading  regulations  and  manuals.  In  addition  to  the  process  description  information,  the 
collected  data  also  included  user  ideas  about  what  is  wrong  with  the  current  process  and  how  it 
could  be  improved. 

The  strategy  for  the  data  collection  effort,  as  shown  in  Figure  D-2,  used  a  set  of  interview 
criteria  established  to  guide  the  development  of  interview  kits  for  the  data  collection  efforts. 
The  criteria  included  an  identification  of  job  functions  within  the  ALCs,  the  Select  Set  of 
aircraft  and  components,  the  types  of  maintenance  to  be  performed  on  the  elements  of  the 
“Select  Set”,  and  the  strawman  model.  The  interview  criteria  also  supported  identification  of  the 
specific  roles  to  be  interviewed  at  each  ALC. 

The  information  collected  from  the  ALCs  was  recorded  in  the  form  of  notes  and  tape  recordings. 
This  information  was  then  organized  and  documented  in  the  DCD  database.  As  the  information 
was  analyzed  and  the  "AS-IS"  models  developed,  additional  data  requirements  were  identified. 
This  information  voids  were  filled  by  using  targeted  follow-up  interviews.  This  information  was 
analyzed  and  used  to  refine  and  expand  the  strawman  model  into  the  "AS-IS"  functional  model. 
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Figure  D-2.  Data  Collection  Strategy 


The  initial  data  collection  efforts  focused  on  information  required  to  develop  the  "AS-IS" 
functional  and  data  models.  Later  in  the  ITI-ALC  program,  a  requirement  was  added  to  develop 
and  apply  process  and  simulation  models  to  the  analysis  effort.  As  a  result  of  these  additional 
program  requirements,  the  data  collection  for  the  process  flow  and  simulation  models  was 
addressed  separately  rather  than  as  a  integral  part  of  the  overall  data  collection  effort. 

D.l.l  Strawman  Functional  Model 

A  strawman  model  was  developed  as  the  first  step  in  the  data  collection  effort.  The  strawman 
provided  a  common  reference  point  for  discussing  the  PDM  process  among  all  personnel 
associated  with  the  ITI-ALC  program;  the  structure  for  collecting,  storing  the  collected  data,  and 
analyzing  the  collected  data;  and  the  starting  point  for  developing  the  "AS-IS"  functional  model. 
The  scope,  viewpoint,  and  analysis  objectives  for  the  strawman  model  were  established  through 
the  goals  and  requirements  defined  for  ITI-ALC  program. 

The  strawman  was  initiated  through  a  joint  effort  between  SRA  and  ARINC  personnel,  and 
refined  and  expanded  with  information  gained  from  a  variety  of  sources.  These  sources 
included  the  Joint  Logistics  Systems  Center  (JLSC’s)  Improved  Functional  Baseline  (IFB) 
IDEFO  Model  and  Glossary  dated  October  1993,  the  analysis  of  depot  maintenance  regulations, 
reviews  of  related  programs,  and  subject  matter  experts  available  as  part  of  the  ITI-ALC  team. 
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The  strawman  was  developed  at  the  AO-diagram  level,  and  extended  to  include  A-1  and  A-2 
rtiagramg  which  related  the  ITI-ALC's  PDM  focus  with  the  total  depot  maintenance  process  and 
the  total  Organizational,  Intermediate,  and  depot  maintenance  process,  respectively.  Using 
scenario-based  walkthroughs,  the  model  was  reviewed  with  AL/HRGO  personnel  to  verify  the 
accuracy  of  the  boundaries  within  the  depot  maintenance  to  be  addressed  during  the  ITI-ALC 
program,  thus  setting  the  scope  of  the  entire  program. 

With  the  program  boundary  confirmed,  the  strawman  model  was  reviewed  by  subject  matter 
experts  available  on  the  ITI-ALC  team  to  verify  the  accuracy  of  the  model.  These  reviews  were 
accomplished  through  a  combination  of  readership  cycles  and  scenario-based  walkthroughs. 
The  completed  strawman  model  was  detailed  enough  to  provide  a  common  reference  for  ALC 
personnel  and  the  ITI-ALC  interview  teams  during  the  data  collection  process,  but  general 
enough  not  to  bias  the  interview. 

D.1.2  Identification  of  the  Information  Sources  Discussion 

The  accuracy  and  elfectiveness  of  the  information  collected  to  describe  the  "AS-IS"  view  of  the 
depot  maintenance  process  was  critical  to  the  overall  success  of  the  program.  To  optimize 
success  of  the  data  collection  effort,  the  process  represented  in  Figure  D-3  was  used  to  identify 
the  specific  information  sources.  The  program  objectives  and  the  strawman  model  specified  the 
requirements  for  the  types  of  information  needed,  the  types  of  information  needed  provide 
criteria  for  the  sites  at  which  the  various  data  would  be  collected,  and  the  information  type 
requirement  along  with  the  selected  sites  helped  to  identify  the  specific  individuals,  documents, 
etc.  from  which  the  information  would  be  collected  at  each  ALC.  Knowing  the  specific  sources 
then  helped  to  select  the  most  effective  data  collection  method. 
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Figure  D-3.  Identifying  Data  and  Data  Sources 


105 


D.1.2.1  Information  Types  Discussion 

The  types  of  information  required  to  describe  the  "AS-IS"  depot  maintenance  process  is  driven 
by  the  program  objectives,  with  the  scope  of  the  information  defined  through  the  strawman 
model.  The  program  objective  was  to  perform  a  BPR  analysis  on  the  PDM  f^rtion  of  the  depot 
maintenance  process,  with  a  limited  look  at  component  repair.  To  perform  this  analysis  required 
a  definition  of  the  process  firom  fimctional,  data,  and  operational  perspectives.  Therefore,  it  was 
necessary  to  identify  the  functions  performed  within  depot  maintenance,  the  information  and 
items  used  by  and  produced  by  each  function,  the  interfaces  among  the  functions,  the  business 
rules  describing  the  relationships  and  operational  usage  of  the  data,  the  flow  of  information  and 
products  through  depot  maintenance,  the  dynamic  characteristics  of  each  function  performed, 
and  the  resources  used  to  exercise  each  function. 

To  gain  an  accurate  representation  of  the  current  PDM  process,  with  a  limit  look  at  component 
repair,  a  select  set  of  system  was  developed,  and  listed  in  Table  D-1,  along  with  their  system 
category. 


Table  D-1.  The  Select  Set  of  Systems 


“Select  Set” 

Category 

F-16 

Light  Weight  A/C 

F-15 

Light-Weight  A/C 

F-22 

Light  Weight  A/C 

C-135 

Heavy  Weight  A/C 

C-5 

Heavy  Weight  A/C 

FI  10 

Engine 

FlOO 

Engine 

EF-1 1 1  Tactical  Jamming 
System 

EW 

LANTIRN 

Avionics 

Advanced  Composites 

Structures 

Landing  Gear 

Structures 

Fuel  Controls 

Hydro/Mechanical 

Constant  Speed  Drives 

Hydro/Mechanical 

Generators 

Electro/Mechanical 
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The  strawman  model  provided  the  scope  and  viewpoint  needed  to  focus  in  on  the  information 
associated  with  this  select  set  of  systems  and  also  provided  a  structure  for  collecting  and  storing 
the  data,  along  with  an  initial  identification  for  possible  sources  of  the  data. 

D.1.2.2  Select  Set  Discussion 

The  select  set  of  systems  were  mapped  against  the  five  ALCs  to  produce  system  profiles  for 
each  ALC,  as  represented  in  Table  D-2.  Using  these  profiles,  the  site  at  which  the  information 
for  each  system  within  the  select  set  was  maintained.  This  mapping  allowed  for  the  selection  of 
what  system  maintenance  data  would  be  collected  at  each  ALC. 

Table  D-2.  The  Select  Set  Mapped  to  ALCs 


“Select  Set” 

Category 

ALC(s) 

F-16 

Light  Weight  A/C 

OO-ALC 

F-15 

Light-Weight  A/C 

WR-ALC, 

SM-ALC 

F-22 

Light  Weight  A/C 

SM-ALC 

C-135 

Heavy  Weight  A/C 

OC-ALC 

C-5 

Heavy  Weight  A/C 

SA-ALC 

FllO 

Engine 

OC-ALC 

FlOO 

Engine 

SA-ALC 

EF-1 1 1  Tactical  Jamming 
System 

EW 

WR-ALC 

LANTIRN 

Avionics 

WR-ALC 

Advanced  Composites 

Structures 

SM-ALC, 

OC-ALC 

Landing  Gear 

Structures 

OO-ALC 

Fuel  Controls 

Hydro/Mechanical 

SA-ALC, 

OC-ALC 

Constant  Speed  Drives 

Hydro/Mechanical 

OC-ALC 

Generators 

Electro/Mechanical 

SM-ALC 

D.1.2.3  Source  Identification  Discussion 

With  the  types  of  information  identified  and  the  appropriate  sites  selected,  the  next  step  was  to 
identify  the  specific  sources  from  which  the  information  would  be  collected  at  each  ALC  site. 
These  sources  of  information  focused  on  the  maintenance  personnel  within  each  of  the  ALCs. 
Based  on  the  knowledge  of  the  ITI-ALC  team's  functional  expert  and  the  high-level  perspective 
provided  by  the  strawman  model,  a  list  of  personnel  skill  types,  or  roles,  were  identified  that 
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could  provide  a  majority  of  information  needed.  The  results  of  this  mapping  are  presented  in 
Table  D-3. 


Table  D-3.  Strawman  vs.  Skill  Matrix 


Strawman  Activity 

Performing  “Roles” 

AO.l  Integrate  Work  Requirements 

Planner 

Scheduler 

Aircraft  Manager 

A0.2  Allocate  Resources 

Scheduler 

Aircraft  Manager 

First-Line  Supervisor _ 

A0.3  Acquire  and  Issue  Parts  and  Supplies 

Mechanic/Specialist 

Aircraft  Manager 

Scheduler 

In-House  Supply 

AO  .4  Repair  Engines  and  Components 

Aircraft  Manager 
Mechanic/Specialist 

A0.5  Maintain  and  Repair  Aircraft 

Aircraft  Manager 
Mechanic/Specialist 

A5.1  Select  Task 

Aircraft  Manager 
Mechanic/Specialist 

A5.2  Obtain  Guidance  Material 

Mechanic/Specialist 

A5.3  Order  Parts 

Mechanic/Specialist 

Scheduler 

In-House  Supply 

A5 .4  Perform  Task 

Mechanic/Specialist 

A5.5  Perform  Functional  Check  Flight 

Flight  Test  Pilot 

Inspector 

A5.6  Record  Maintenance  Information 

Aircraft  Manager 

Scheduler 

The  final  step  in  the  source  identification  process  was  to  identify  specific  mdividuals  responsible 
for  the  designated  roles  at  each  ALC.  To  facilitate  this  selection  process,  as  well  as  facilitate  the 
overaU  data  collection  effort,  a  Point-of-Contact  (POC)  was  arranged  at  each  site.  Biographies 
were  developed  and  provided  to  the  focal  point  at  each  ALC.  This  biographies  provided  a 
description  of  the  skills  that  had  been  identified  as  information  sources.  Usmg  these  biograpmes 
as  a  guide,  the  POC  selected  specific  individuals  who  could  effectively  provide  the  required 
information  and  to  arranged  for  their  time  during  the  scheduled  data  collection  trip. 

As  the  data  collection  effort  progressed,  supplementary  information  sources  such  as  forms, 
reports,  regulations,  process  observation  were  identified  and  used. 
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D.1.3  Data  Collection  Materials 


With  the  information  collection  goals  established  along  with  the  initial  definition  of  the 
information  sources,  the  next  step  was  to  establish  how  the  information  wotild  actually  be 
collected.  The  data  collection  material  that  were  developed  were  interview  kits,  the  data 
collection  team  organization,  and  data  collection  procedures  definition. 

D.13.1  Interview  Kits 

The  objective  of  the  Interview  Kits  was  to  provide  all  the  information  needed  the  support  the 
data  collection  effort.  An  interview  kit  contained  the  following  materials: 

•  Section  A:  Interviewer’s  Guide 

A  outline  defining  the  general  steps  to  be  accomplished  during  the  interview. 

•  Section  B:  Introduction 

An  overview  of  the  ITI-AJLC  program  that  provides  the  interviewee  with  a 
description  of  the  program  for  which  the  information  is  being  collected. 

•  Section  C:  Privacy  Act  Signature  Collection  Form 

A  agreement  that  is  signed  and  dated  before  an  interview  can  proceed.  This 
agreement  assures  that  the  interviewee  is  participating  in  a  voluntaiy  manner  and 
that  all  information  provided  will  not  be  documented  in  a  maimer  that  links  the 
information  back  to  the  individual. 

•  Section  D:  Biography 

A  form  which  documents  the  title  and  skill  level  of  the  individual  when  an  interview 
approach  is  applied. 

•  Section  E:  Results  Package  Cover  Sheets 

A  form  which  links  a  unique  number  to  the  interview  along  with  the  identification  of 
the  interviewers. 

•  Section  F:  Strawman  Model 

The  IDEFO  strawman  model,  along  with  its  associated  text.  This  strawman  model  is 
used  as  the  focal  point  for  discussing  the  process  being  analyzed. 

•  Section  G;  Interview  Numbers  (allocated  according  to  the  ALCs  and  interview 
team) 

A  set  of  interview  numbers  assigned  to  each  data  collection  team.  For  the  ITI-ALC 
program,  this  list  included  a  large  set  for  each  ALC.  This  large  set  for  the  ALC  was 
then  subdivided  into  subsets  and  assigned  to  each  team. 

•  Section  H:  Select  Set  of  Reparables  per  the  ITI-ALC  Proposal 

A  listing  of  the  select  set  of  reparables  to  be  focused  on  at  each  ALC. 
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•  Section  I:  Interview  Breakout  Schedule 

A  table  which  summarizing  the  administrative  mformation  related  to  each  interview. 
This  table  was  used  to  schedule  the  planned  interviews  and  to  document  variations 
to  the  schedule. 

•  Section  J:  Question  Sets 

A  set  of  questions  diat  were  developed  and  used  to  guide  the  data  collection  effort. 
These  questions  were  not  meant  to  be  used  in  a  direct  question  and  answer 
(questionnaire)  manner,  but  rather  as  a  flexible  guide  or  check  point  to  ensure  dl 
information  was  collected  during  the  session.  The  question  set  contained  generic 
questions  that  were  applied  across  all  nodes  of  the  strawman  model  as  well  as 
specific  questions  were  developed  that  targeted  areas  not  well  understood  by  the  ITI- 

ALCteam. 

•  Section  K:  Acronyms 

A  list  of  acronyms  previously  identified  as  being  used  within  the  process  being 
analyzed.  During  the  data  collection,  these  acronyms  were  verified  and  new  ones 
as  they  were  identified.  They  provided  a  quick  reference  for  the  data 
collection  team. 

•  Section  L:  Data  Collection  Form 

Blank  pages  on  which  the  data  collection  team  could  take  notes  during  the  data 
collection  sessions. 

•  Section  M:  In-Briefing  (developed  by  AL/HRGO) 

A  set  of  briefing  charts  used  as  the  first  step  at  each  ALC  to  present  the  ITI-ALC 
program  and  the  objective  of  the  data  collection  effort. 

•  Section  N:  Miscellaneous  0«gacy  system  definitions,  interview  team  composition, 
etc.) 

Blank  pages  on  which  the  data  collection  team  could  take  notes  about  legacy 
systems  and  other  information  that  could  be  of  value  in  the  program  but  not 
necessarily  within  the  boundaries  of  the  program. 

There  exists  a  number  of  data  collection  procedures  that  could  have  been  used  to  gather  the 
necessary  information  for  the  ITI-ALC  program  firom  the  users  and  functional  exerts.  These 
techniques  include  questionnaires,  group  workshops,  observation,  documentation  review,  and  user 
interviews.  For  the  m-ALC  program,  all.  these  approaches  except  for  the  questionnaires  were 
used  to  some  degree,  but  the  primary  approaches  were  user  interviews  and  operational 

observations. 

The  documentation  reviews  were  used  to  gain  an  initial  understanding  of  the  depot  maintenance 
process  and  to  analyze  the  documents  and  forms  collected  firom  the  ALCs.  These  data 
supplemented  data  firom  other  sources. 
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Guided  by  the  interview  kits,  many  of  the  interviews  were  held  away  from  the  interviewee's 
worksite  so  as  to  allow  the  interviewee  to  focus  on  the  interview  and  not  be  distracted  by  daily 
activity  going  on  within  the  depot.  The  remaining  interviews  were  held  on-site  to  provide  a  first¬ 
hand  understanding  and  appreciation  of  the  depot  maintenance  work  performed  and  the  actual 
access  and  use  of  information. 

The  observations  were  implemented  primarily  in  the  form  of  facility  tours  that  allowed  for  direct 
interaction  with  the  users.  In  addition,  a  number  of  visits  were  made  to  the  specific  areas  within 
the  depot  to  observe  the  work  performed  by  specific  individuals  who  played  a  critical  role  in  the 
depot  maintenance  process. 

Using  walkthroughs,  workshops  were  used  primarily  to  validate  the  models.  However,  during 
these  workshops,  additional  depot  maintenance  process  data  was  also  collected  and  used  to 
refine  and  expand  the  various  models. 

D.1.3.2  Data  Collection  Teams 

The  ITI-ALC  team’s  emphasis  on,  and  sensitivity  to,  the  need  for  accurate  data  collection  and 
analysis  demanded  that  the  data  collection  structure  and  teams  be  developed  very  carefully. 
Each  data  collection  effort  was  planned  so  as  to  address  the  data  required  to  be  collected  from 
the  both  the  site  and  the  specific  source. 

A  key  element  in  ensuring  the  accuracy  and  completeness  of  the  data  collection  was  to  establish 
an  effective  set  of  data  collection  teams.  Each  team  was  limited  to  two  Individuals;  a  modeling 
expert  and  a  functional  area  expert.  The  presence  of  too  many  interviewers  on  a  team  has  a 
tendency  to  intimidate  and  confuse  a  subject,  resulting  in  invalid  data. 

The  primary  responsibility  of  the  modeling  experts  was  to  guide  the  interview  as  specified  in  the 
interview  Idts  so  as  to  ensure  that  data  collected  was  sufficient  for  the  development  of  the 
various  models.  The  primary  responsibilities  of  the  functional  area  experts  were  to  ensure  the 
(iata  collected  encompassed  all  aspects  of  the  depot  maintenance  process,  to  ensure  the  modelers 
were  correctly  interpreting  the  subject's  information,  to  ask  appropriate  questions  to  ensure 
information  accuracy  and  completeness,  and  to  act  as  a  sounding  board  for  discussing  the 
modeler's  understanding  of  the  maintenance  process. 

As  a  member  of  the  overall  data  collection  team,  but  not  specifically  assigned  as  a  member  of  a 
two-person  data  collection  team,  a  observer  had  the  freedom  to  attend  any  of  the  interview 
sessions.  The  responsibility  of  the  observer  was  not  to  take  an  active  part  in  the  interview  but 
rather  to  monitor  all  the  data  collection  teams  to  ensure  consistency  of  data  collection  across  all 
teams. 

D.1.3.3  Site  Visit  Strategy 

For  data  collection  visit  to  an  ALC  started  with  an  in-briefing  and  tour.  This  was  followed  with 
scheduled  sets  of  interviews  and  some  process  observations  when  possible  and  appropriate.  At 
the  end  of  the  week  an  out-briefing  was  offered  to  the  selected  personnel  at  the  ALC. 
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For  each  interview,  the  strawman  model  was  iised  as  the  basis  for  discussion  and  provided  two 
main  functions.  One,  it  demonstrated  to  the  subjects  the  level  of  knowledge  possessed  by  the 
data  collection  team  and  provided  the  subject  with  a  reference  point  at  which  to  direct  their 
discussion.  Two,  the  strawman  model  provided  a  process  boundary  and  therefore  helped  to 
focus  and  direct  the  data  collection  effort. 

During  each  interview,  data  was  collected  in  a  nmnber  of  forms.  Notes  were  taken  either 
electronically  or  handwritten,  depending  on  the  preference  of  the  team.  Each  interview  was  also 
recorded  with  a  audio  tape,  widi  the  subject  being  given  the  opportunily  to  declined  having  the 
recording  made  or  to  stop  the  recording  at  any  point  during  the  interview.  As  references  were 
madf.  to  current  information  systems  and  data  recording  forms,  information  about  the  systems 
and  copies  of  the  forms  were  requested  and  obtained.  A  list  of  these  information  systems  and 
forms  was  maintained  among  the  data  collection  teams  to  provide  a  point  of  reference  and  to 
identify  that  information  had  already  been  requested  and  or  received  about  the  systems  and 
forms. 

Following  each  interview,  the  team  would  meet  to  discuss  the  process  understanding  gained 
from  the  interview,  to  identify  specific  information  requirements  that  must  be  addressed  during 
subsequent  d?*ta  collection  efforts,  and  to  organize  and  enter  the  information  into  the  DCD. 

At  the  end  of  each  day,  the  data  Collection  teams  met  to  discuss  their  data  collection  efforts  for 
the  day.  These  discussion  included  new  rmderstandings  that  had  been  gained,  information  voids 
that  had  been  identified,  and  specific  data  collection  requirements  for  the  next  day. 

D.1.4  Managing  the  Collected  Data 

A  significant  effort  was  required  to  collect  the  information  from  the  users  and  other  sources  for 
its  use  in  understanding,  analyzing,  and  developing  the  improved  depot  maintenance  operational 
concept.  To  maximize  the  benefits  received  from  the  information  required  that  the  information 
be  effectively  managed. 

Shortly  after  each  data  collection  interview,  the  team  organized  the  information  and  entered  it 
into  the  DCD.  There  were  two  reasons  for  entering  the  data  as  soon  as  possible.  First,  the 
sooner  the  information  was  entered  into  the  DCD,  the  sooner  the  information  was  available  for 
use  in  refining  and  expanding  the  models.  Second,  the  sooner  the  information  was  entered  into 
the  DCD  the  completeness  and  accuracy  of  the  information  was  increased  due  primarily  to  the 
memory  and  note  taking  capability  of  the  interviewers. 

The  information  for  each  interview  was  identified  by  an  interview  number  which  related  it  back 
to  the  ALC  and  the  skill  from  which  it  was  collected.  From  the  collected  information,  activities 
were  identified,  short  narratives  written  about  each  activity,  and  the  inputs,  outputs,  controls, 
and  mechanisms  for  the  activities  identified.  In  addition,  the  interactions  among  various 
activities  were  also  identified. 
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To  reduce  the  duplication  of  effort  during  the  data  entry  effort,  a  standardized  list  of  activities 
were  maintained  in  the  DCD.  As  an  activity  was  identified  for  the  collected  data,  a  quick  search 
was  made  to  determine  if  that  activity  had  been  previously  entered.  If  the  activity  did  exist,  the 
associated  information  was  reviewed  and  updated  as  necessary.  If  the  activity  did  not  exist,  the 
appropriate  information  would  be  entered  into  the  DCD. 

The  tapes  were  labeled  with  the  interview  number,  notes  were  placed  in  a  folder  labeled  with  the 
interview  number,  and  the  forms  were  organized  in  a  three-ring  binder  with  an  form  index 
placed  in  front  for  a  quick  reference. 

D.1.5  Tool  Selection 

The  tools  used  to  support  the  data  collection  process  were  the  interview  kit,  a  tape  recorder  and 
the  Data  Collection  Device. 

D.1.5.1  Interview  Kit 

The  interview  kit  was  implemented  as  a  manrral  tool  to  provide  guidance  and  support  during  the 
data  collection  effort.  This  kit  as  described  in  Section  D.  1.3.1  included,  among  other 
information,  the  t3q)es  of  information  to  be  collected  and  a  means  of  collecting  notes  during  the 
data  collection  efforts. 

D.1.5.2  Tape  Recorder 

During  a  data  collection  session,  significant  amounts  of  information  are  presented.  However, 
due  to  a  number  of  factors,  much  of  this  information  could  not  be  fully  imderstood  or 
appreciated  at  the  time,  or  could  not  be  retained  following  the  data  collection  effort.  Therefore, 
to  ensure  the  full  potential  of  the  data  collection,  tape  recorders  were  used  to  capture  everything 
that  was  said.  These  tapes  then  provided  a  valuable  resource  to  revisit  the  data  collection  efforts 
over  a  period  of  time  so  that  the  full  value  of  the  interviews  was  acquired. 

D.1.5.3  The  Data  Collection  Device 

The  DCD  was  a  data  base  tool  used  to  support  the  data  collection  and  model  development 
efforts.  The  DCD  was  developed  using  COTS  data  base  management  system  (DBMS)  4th 
Dimension  by  ACI  US,  Inc.  as  an  engine  and  to  run  on  a  portable  computer  that  could  easily  be 
taken  on  the  data  collection  trips.  The  DCD  was  capable  of  coordinating  the  collected  data  with 
the  structure  of  the  strawman  model,  capturing  all  of  the  data  anticipated  to  be  gathered  during 
data  collection,  and  allowing  for  data  entry  and  access  in  a  way  that  was  intuitive  to  the  data 
collectors  and  model  developers. 
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